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EXECUTIVE  SUMMARY 


1 .  INTRODUCTION 

A  program  was  undertaken  to  develop  procedures  for  analyzing  and  predicting  the 
testability  attributes  of  systems.  This  report  describes  the  results  of  the 
first  phase  of  that  program.  The  initial  phase  has  concentrated  on  the  three 
testability  attributes  most  commonly  specified  for  systems: 

Fraction  of  faults  detected  (FFD) 

Fraction  of  faults  Isolated  (FFI) 

Fraction  of  false  alarms  (FFA) 

These  attributes  were  defined  and  scoped  and  mathematical  frameworks  have  been 
developed  for  evaluating  each  attribute  at  the  organizational  level  of 
maintenance.  Further,  the  feasibility  of  developing  prediction  procedures  for 
these  three  attributes  was  investigated. 

The  definitions  and  mathematical  frameworks  were  developed  through  the 
application  of  three  different  approaches  to  modeling  the  organizational  level 
maintenance  process.  The  three  modeling  approaches  employed  were: 

Set  Theory  Model  -  model  was  developed  through  the  use  of  Venn  diagrams 
and  set-membership  approaches  to  derive  definitions  and  algorithms. 

Modified  State  Model  -  model  based  upon  the  combination  of  actions  at  the 
organizational  maintenance  level  necessary  to  discover  the  system  state 
(e.g.,  failed  or  non-failed). 

Flow  Model  -  model  that  traces  the  flow  of  systems  and  subsystems  through 
the  organizational  level  maintenance  process. 

The  use  of  the  different  approaches  had  several  advantages;  two  of  these 
advantages  were: 

(1)  The  insights  and  visibility  into  the  interpretation,  make-up  and  logical 
content  of  an  attribute  afforded  through  the  use  of  one  modeling  approach 
were  often  superior  to  those  provided  by  another.  Further,  the  different 
viewpoints  provided  by  each  of  the  modeling  approaches  combined  to 
provide  insights  into  the  form  and  content  of  the  attributes  that  could 
not  have  been  provided  by  the  application  of  a  single  model.  And, 

(2)  The  use  of  the  three  modeling  approaches  provided  a  means  for  cross¬ 
checking  the  results  of  the  models.  Because  all  three  approaches  model 
the  organizational  level  maintenance  process,  the  three  models  must 
provide  consistent  results. 

This  report  describes  the  models  and  algorithms  used  to  develop  the  definitions 
and  evaluation  procedures  for  the  three  testability  attributes:  fraction  of 
faults  detected,  fraction  of  faults  isolated  and  fraction  of  false  alarms.  The 
literature  research  and  survey  of  organizational  level  maintenance  units  that 
contributed  to  the  development  of  the  definitions  of  the  testability  attributes 
are  provided  in  Chapters  2  and  3  of  this  report.  The  sources  and  types  of 
maintenance  data  that  are  available  from  operational  systems  that  can  be  used  to 


measure  the  testability  attributes  are  discussed  in  Chapter  4.  The  actual 
definitions  and  evaluation  algorithms  developed  from  the  three  models  are 
presented  in  Chapter  5.  The  feasibility  of  developing  procedures  for  predicting 
the  three  testability  attributes  is  discussed  in  Chapter  6.  Finally,  the  results 
of  the  first  phase  of  this  program  are  summarized  and  plans  for  the  development 
of  prediction  procedures  during  the  second  phase  of  the  program  are  presented  in 
Chapter  7.  Detailed  descriptions  and  mathematical  analyses  of  the  three  modeling 
approaches  are  provided  in  the  appendices. 

The  work  that  is  presented  in  this  report  Is  the  foundation  for  continuing 
research.  As  such,  this  report  is  intended  to  be  a  working  draft,  and  the 
reader's  comments  on  the  usefulness  and  possible  applications  of  the  work 
presented  in  this  report  are  welcomed.  Comments  may  be  addressed  to  Rome  Air 
Development  Center,  RADC/RBET  (H.  Dussault),  Griffiss  AFB  NY  13441-5700. 

A  brief  summary  of  the  definitions  developed,  modeling  analyses,  and  results  of 
the  feasibility  study  follow. 


2.  PROPOSED  DEFINITIONS  AND  THEIR  LOGIC 

A  key  objective  of  the  first  phase  of  the  program  was  to  develop  accurate, 
quantitative  definitions  of  the  three  testability  attributes.  The  definitions 
to  be  developed  were  to  be  relevant,  consistent  with  military  standards,  math¬ 
ematically  precise,  and  measurable.  It  would  be  of  little  use  to  derive  a  set  of 
equations  that  could  not  be  used  to  measure  fielded  system  attributes.  Toward 
that  end,  two  realistic  assumptions  were  made. 

(1)  Any  fault  indication  that  does  not  result  in  a  maintenance  action  is  a 
nonrelevant  event.  Under  most  definitions  of  system  behavior,  these 
events  are  called  false  alarms  that  are  recognized  but  ignored  'i.e.,  not 
reported).  These  events  are  totally  unmeasurable  and  have  little  impact 
on  the  maintenance  system. 

(2)  Faults  that  are  not  literally  detected  by  any  means  are  nonrelevant  on 
the  basis  that  they  have  no  discernable  impact  on  and  are  not  measurable 
at  the  organizational  level  of  maintenance. 

Before  proceeding  further  with  the  definitions  of  the  testability  attributes  for 
the  system  and  subsystem  levels,  it  is  necessary  to  define  normal  system 
maintenance  as  it  Is  used  in  the  development  of  definitions  for  each  of  the 
testability  attributes. 

Normal  System  Maintenance  (NSM)  -  Techniques  that  are  specified  as  standard 
operating  procedures  for  use  of  BIT,  ATE,  semiautomatic,  or  documented 
manual  detection  and  troubleshooting  for  a  given  system  under  test.  They 
include  regular  calendar  checks  and  normal  "go"  checks.  NSM  is  sometimes 
called  "defined  means".  (1) 


(1)  RADC  Testability  Notebook,  Hughes  Aircraft  Company, 
RADC-fR-82- 189,  June  1982. 


The  definitions  of  systems  and  subsystems  must  also  be  addressed  before 
definitions  of  the  testability  attributes  can  be  developed.  The 
system/subsystem  boundary  is  an  artificial  one  and  is  drawn  on  the  basis  of 
analysis  needs.  A  system  is  taken  as  a  functional  or  structural  entity.  Its 
boundaries  are  often  physical  breakpoints  between  the  system  and  its  surrounding 
environment.  For  the  purposes  of  this  effort  the  boundaries  between  systems  and 
subsystems  are  defined  from  the  analyst's  perspective,  as  may  be  demonstrated  by 
the  following  example.  The  government,  concerned  with  the  acquisition  of  weapon 
system  XYZ,  would  consider  weapon  system  XYZ  to  be  the  system  and  LRU  5  to  be  a 
subsystem  of  XYZ.  However,  a  contractor  who  builds  LRU  5,  and  only  LRU  5  of 
weapon  system  XYZ,  would  consider  LRU  5  to  be  a  system. 

The  following  definitions  have  been  developed  for  system  level  fault  detection, 
fault  isolation,  and  false  alarms: 

Fault  detection  -  NSM  indicates  that  the  system  is  not  functioning 
properly,  and  this  indication  is  the  result  of  a  fault  within  the  system 

Fault  isolation  -  NSM  identifies  all  failed  units  within  the  system. 
Fault  isolation  may  be  either  proper  or  improper. 

Proper  fault  isolation  -  Only  and  all  failed  units  are 
isolated. 

Improper  fault  isolation  -  All  but  not  only  failed  units  are 
isolated. 

Note:  Any  other  outcome  of  an  attempted  isolation  is  considered  to 
result  in  No  Fault  Isolation. 

False  alarm  -  There  is  an  indication  of  failure  in  the  system  where  there 
is  no  failure  in  the  system.  False  alarm  rate  (FAR)  is  the  sum  of  false 
alarms  over  a  general  time  period  divided  by  that  time  period. 

The  system  definitions  must  also  be  consistent  with  subsystem  definitions  in  the 
hierarchical  sense  as  subsystems  are  built  into  systems.  The  consistency  of  the 
definitions  requires  that  the  system/subsystem  boundary  be  defined  in  advance  of 
any  analysis.  The  following  definitions  have  been  developed  for  subsystem  level 
fault  detection,  fault  isolation,  and  false  alarms. 

Fault  detection  -  NSM  indicates  that  a  subsystem  is  not  functioning 
properly,  because  of  a  fault  within  the  system.  The  detection  can  be 
proper  or  improper. 

Proper  detection  -  The  fault  is  within  the  subsystem  in  which 
the  detection  occurs. 

Improper  detection  -  The  fault  is  within  a  subsystem  other 
than  the  one  in  which  the  detection  occurs. 


Fault  isolation  -  NSM  identifies  all  failed  units  within  a  subsystem. 
The  isolation  can  be  proper  or  improper. 

Proper  isolation  -  Only  and  all  failed  units  are  isolated. 

Improper  isolation  -  All  but  not  only  failed  units  are 
isolated. 

Note:  Any  other  outcome  of  an  attempted  isolation  is 

considered  to  result  in  No  Fault  Isolation. 

False  Alarm  -  There  is  an  indication  of  failure  in  the  subsystem  where 
there  is  no  failure  in  the  system. 

Fraction  of  faults  detected  (FFO)  and  fraction  of  faults  isolated  ( FFI )  are 
derived  by  dividing  the  total  system  or  subsystem  detection  and  isolation  values 
by  the  total  faults  in  the  system  or  subsystem.  Fraction  of  false  alarms  (FFA) 
is  derived  by  dividing  the  false  alarm  total  by  the  total  number  of  maintenance 
actions  either  at  the  system  or  subsystem  level.  The  false  alarm  rate  (FAR)  is 
derived  by  dividing  the  false  alarm  total  by  the  time  period  over  which  those 
false  alarms  developed. 


3.  MODELING  SUMMARY 

Three  representations  of  the  organizational  level  maintenance  process  were 
derived.  These  three  models  were  based  on  set  theory,  modified  state,  and  flow 
model  assumptions,  as  discussed  previously. 

All  three  models  agree  on  functionality.  For  example,  in  each  of  the  models,  FFA 
is  a  function  of  "cannot  duplicate"  results  and  maintenance  actions.  The  form  of 
all  key  parameters  is  identical.  Further,  each  model  points  out  that  it  is 
important  to  know  what  triggers  maintenance  activity  and  how  fault  isolation  is 
achieved.  The  current  Air  Force  maintenance  reporting  system,  however,  does  not 
provide  complete  information  on  actions  that  trigger  maintenance  ( e . g . ,  BIT 
report,  pilot  report)  or  what  actions  were  used  to  achieve  fault  isolation  (e.g., 
tech  orders,  ad  hoc  "shotgun"  approaches).  The  models,  therefore,  point  out 
limitations  in  measuring  the  three  testability  attributes  based  on  field 
reported  data.  Each  model  also  points  out  the  difficulty  in  relating  "cannot 
duplicate"  results  to  false  alarms  and  the  importance  of  measuring  false  alarms. 
In  every  case,  the  accurate  evaluation  of  FFD  and  FFI  requires  an  accurate 
measurement  of  false  alarms. 

As  mentioned  previously,  each  of  the  models  highlights  separate  insights  into  the 
measurement  and  analysis  of  system  testability  attributes.  The  set  theory  model 
forces  an  explicit  statement  of  assumptions  that  are  inherent  in  all  three  models 
but  not  explicitly  stated.  The  set  theory  model  also  provides  a  method  for 
specifying  what  should  be  measured.  The  flow  model  representation  provides  a 
direct  link  between  the  maintenance  model  and  readiness  and  shows  the  limits  that 
must  be  placed  on  data  gathering  in  terms  of  time  sufficiency,  periodicity  and 
quantity  of  data.  Because  of  its  inherent  simplicity  and  conformity  with  current 
maintenance  data  gathering,  the  modified  state  model  represents  the  best 
computational  fit. 


4.  FEASIBILITY  SUMMARY 


The  feasibility  of  developing  prediction  procedures  for  the  three  testability 
attributes  was  determined  based  upon  two  major  criteria:  1)  the  ability  to 
measure  FFD,  FFI ,  and  FFA  in  currently  fielded  systems,  and  2)  the  ability  to 
relate  specific  design  parameters  to  measured  values  of  the  testability 
attributes.  If  both  of  these  criteria  could  be  satisfied,  the  development  of 
prediction  procedures  would  be  considered  feasible. 

As  has  been  discussed  previously,  field  measurement  of  the  three  testability 
attributes  of  interest  is  difficult.  The  current  Air  Force  maintenance  data 
collection  system  does  not  provide  direct  measures  of  FFD,  FFI,  or  FFA.  The 
maintenance  data  collection  system  does  record  "cannot  duplicate"  events,  and  a 
measurement  of  "cannot  duplicate"  events  and  maintenance  actions  could  be  used  to 
derive  a  measure  of  false  alarms.  The  field  measurement  of  FFD  and  FFI,  however, 
requires  direct  observation  of  what  triggered  the  maintenance  activity  and  how 
fault  isolation  was  achieved.  Other  measures  of  FFD  and  FFI  could  be  derived 
using  system  design  information,  maintainability  demonstration  and  operational 
test  and  evaluation  data,  and  testability  modeling  and  analysis  data. 

Establishing  relationships  between  system  design  parameters  and  the  testability 
attributes  should  be  feasible  once  measures  of  the  attributes  can  be  obtained. 
These  parameters  include:  number  of  elements,  number  of  test  points,  number  of 
feedback  loops,  degree  of  parallelism  in  the  design,  and  connector  dependency. 
An  investigation  of  possible  relationships  between  the  design  parameters  and  the 
testability  attributes  was  conducted  using  a  limited  data  set  and  only  one 
testability  attribute,  FFA.  The  preliminary  results  of  the  investigation 
indicate  that  a  relationship  exists  between  the  degree  of  parallelism  in  a  given 
design  and  the  number  of  false  alarms  experienced  by  the  system.  In  general,  the 
feasibility  of  developing  the  relationships  appears  to  be  promising. 

The  continuation  of  the  current  work  toward  developing  prediction  procedures 
requires  that  the  difficulties  in  measuring  the  three  testability  attributes  be 
overcome.  Two  different  approaches  are  required  to  obtain  the  information 
necessary  to  develop  measures  of  the  three  testability  attributes.  First,  field 
data  on  maintenance  actions  and  "cannot  duplicate"  events  will  be  gathered  for  a 
number  of  LRUs.  Measures  of  false  alarms  can  be  analytically  or  heuristically 
determined  from  this  field  data.  Second,  measures  of  fractions  of  faults 
detected  and  fraction  of  faults  isolated  will  be  derived  from  engineering  and 
field  test  data  (e.g.  maintainability  demonstrations  and  technical  and 
operational  evaluations)  and  the  application  of  testability  models  and  analyses 
(e.g.  FMEAs).  Once  measures  of  FFD,  FFI,  and  FFA  have  been  developed, 
prediction  procedures  will  be  developed  by  relating  the  testability  attributes 
to  system  design  parameters. 
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INTRODUCTION  AND  BACKGROUND 


1 . 1  INTRODUCTION 

This  report  describes  research  conducted  by  ARINC  Research  Corporation 
in  the  first  phase  of  a  two-phase  project.  The  work  was  performed  under 
Contract  F30602-84-C-0046  with  the  Systems  Reliability  and  Engineering 
Branch  of  the  Rome  Air  Development  Center. 

The  ultimate  goal  of  the  research  is  to  build  a  model  that  will  pre¬ 
dict  organizational-level  testability  attributes  on  the  basis  of  design 
characteristics.  Three  oaslc  descriptors  of  the  organizational-level 
maintenance  system  were  considered: 

-  Fraction  of  faults  detected  (FFD) 

-  Fraction  of  faults  isolated  (FFI) 

-  Fraction  of  false  alarms  (FFA) 

The  Phase  I  technical  objectives  were  to  provide  the  foundation  for 
the  development  of  the  predictor  model.  The  approach  was  developed  through 
the  following  tasks: 

-  Survey  the  current  literature  and  the  personnel  engaged  in 
organizational-level  maintenance. 

-  Compile  and  define  location  and  types  of  data  resources  currently 
available. 

-  Develop  a  consistent  mathematical  structure  that  will  permit  the 
measurement  of  the  required  parameters  and  the  development  of  con¬ 
sistent  definitions. 

-  Determine  the  feasibility  of  developing  useful  prediction  methods 
and  identify  the  approaches  necessary  for  such  development. 


1.2  BACKGROUND 


As  a  result  of  Increased  system  complexity  and  sophistication,  the 
maintenance  of  electronic  systems  is  becoming  more  difficult  and  costly. 

despite  advances  in  automatic  test  equipment.1'3  Testability  design  is 
usually  approached  from  the  bottom  up,  with  component  and  board  testa¬ 
bility  designed  In  but  with  little  attention  given  to  Isolation  to  the 
Individual  unit  In  the  full  system.  Current  design  of  systems  and  tests 
frequently  results  In  long  test  times  and  high  ambiguity  levels  for  fault 
Isolation.  False-alarm  and  M retest -OKM  (RTOK)  rates  of  40  percent  and 
greater  are  not  uncommon  In  many  avionic  systems.  Studies  of  the  F-16 

aircraft3  and  the  CH-54  helicopter4  have  shown  that  troubleshooting 
can  consume  50  percent  or  more  of  the  total  man-hours  expended  on  repair. 
Avionics  Maintenance  Conference  reliability  reporting  statistics  indicate 
similar  trends  in  avionic  repairs  for  the  scheduled  air  carriers.  Those 
figures  suggest  the  potential  for  a  large  return  on  an  Investment  In  im¬ 
proved  testability  assessment  leading  to  Improved  testability  design. 

1.2.1  The  Testability  Discipline 

Testability  Is  coming  to  be  recognized  as  a  valid  and  useful  engi¬ 
neering  discipline.  The  recent  Issuance  of  a  testability  standard.  Testa¬ 
bility  Program  for  Electronic  Systems  and  Equipments. *  is  evidence  of  the 
Increasing  Importance  of  testability  in  the  development  of  military 
systems.  An  equipment  has  good  testability  when  existing  faults  can  be 
confidently  and  efficiently  Identified.  Confidence  Is  achieved  by  fre¬ 
quently  and  unambiguously  Identifying  only  the  failed  components  or  parts, 
with  no  removals  of  good  Items  and  with  minimum  loss  of  time  due  to  false 
Indications  or  false  alarms.  Efficiency  Is  achieved  by  minimizing  the 
resources  required,  such  as  man-hours,  test  equipment,  and  training. 

1.2.2  Testability  as  a  Design  Variable 

The  number  of  tests  and  the  information  content  of  test  results, 
together  with  the  location  and  accessibility  of  test  points,  define  the 
testability  potential  of  an  equipment.  Testability  Is,  of  course,  a 
design- related  characteristic.  There  are  few  standardized  tools  for  the 
evaluation  of  design  testability,  particularly  at  the  organizational  level 
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In  fact,  a  review  of  the  current  literature  suggests  that  even  common 

definitions  of  testability  are  hard  to  find.  Por  example.  Malcolm1  states 
that  built-in-test  (BIT)  false  alarms  can  be  broken  Into  two  types: 
a  BIT  Indication  when  there  are  no  faults,  and  a  BIT  Indication  when  the 

fault  Is  In  another  unit.  MIL-STD-13093  defines  a  false  alarm  as  a  fault 
Indication  where  no  fault  exists.  Whether  these  two  definitions  are 
consistent  depends  on  Individual  Interpretation. 

For  testability  to  be  appropriately  and  consistently  Incorporated 
Into  the  design  process,  standard  definitions,  procedures,  and  tools  must 
be  developed  to  evaluate  and  predict  organizational-level  testability 
attributes.  A  testability  evaluation  should  provide  not  only  predictions 
but  also  applicable  redesign  information  when  testability  attributes  are 
predicted  to  be  below  desired  levels. 


1.2.3  Testability  and  Organizational-Level  Maintenance 

The  problem  of  testability  at  the  organizational  level  Is  separate 
from  but  related  to  the  same  problem  at  the  Intermediate  and  depot  levels. 
The  organizational  level  Is  where  system  faults  are  first  detected.  The 
Interaction  of  subsystems  complicates  fault  Identification  and  detection. 
Organizational-level  testability  Is  a  primary  Influence  on  mission  readi¬ 
ness.  and  lack  of  fault  detection  at  this  level  can  lead  to  mission  fail¬ 
ure.  Of  the  many  testability  attributes  that  we  will  explore,  three  are 
directly  related  to  the  ability  of  complex  electronic  systems  to  meet  mis¬ 
sion  requirements: 

-  Fraction  of  Faults  Detected  (FFD)  -  Ideally.  FFD  should  be  100 
percent.  Any  fault  not  detected  prior  to  a  mission,  either  by 
BIT/BITE  or  by  maintenance  operations  ready  (OPSREADY)  test,  could 
result  in  a  failed  or  aborted  mission.  Further,  If  the  failure  is 
not  detected  after  the  mission,  the  following  mission  could  be 
jeopardized.  In  reality,  some  system  faults  are  less  critical 
than  others,  and  an  FFD  smaller  than  100  percent  might  be 
tolerable. 

-  Fraction  of  Faults  Isolated  (FFI)  -  The  ideal  value  of  FFI  is  100 
percent.  If  a  detected  failure  Is  not  Isolated  quickly  and  effi¬ 
ciently.  the  system  may  not  be  mission-ready  for  a  long  time.  To 
meet  the  mission-ready  requirement,  maintenance  crews  may  change 
out  entire  mission-critical  systems  or  spend  a  great  deal  of  time 
using  “shotgun"  maintenance  approaches.  These  practices  complicate 
already  difficult  sparing  and  logistics  problems  and  add  to  a  sys¬ 
tem's  life-cycle  costs.  Measures  associated  with  FFI  are  mean 
time  to  fault-lsolate  (MTFI)  and  mean  time  to  repair  (MTTR) ,  as 
well  as  ambiguity  group  statistics  and  RTOK  rates. 
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-  Fraction  of  False  Alarms  (FFA)  -  The  Ideal  value  of  FFA  Is  0  per¬ 
cent;  FFA  Is  a  complementary  factor  of  FFD.  When  BIT/BITE  or 
OPSREADY  checks  Indicate  failures  that  cannot  be  duplicated  or 
Isolated  because  they  do  not  exist,  the  system  Is  held  from 
mission-ready  status  while  checks  are  run  and  rerun.  A  high  FFA. 
like  a  low  FFI,  leads  to  system  change-outs  or  "shotgun"  mainte¬ 
nance  approaches. 

A  related  parameter  that  will  only  be  dealt  with  peripherally  is 
false-alarm  rate  (FAR),  the  rate  of  occurrence  of  false  alarms.  It  Is 
typically  computed  as  the  time-normalized  sum  of  false  alarms,  where  the 
time  normalization  Is  either  calendar  or  operating  hours. 


1.3  REPORT  ORGANIZATION 

Chapter  Two  of  this  report  provides  details  of  the  literature  search 
and  a  compilation  of  prior  efforts  in  this  area.  Chapter  Three  briefly 
outlines  the  organizational-level  personnel  survey  and  Its  results.  Chap¬ 
ter  Four  delineates  data  resources  developed  for  use  with  this  study. 

Chapter  Five  Is  an  explanation  of  the  measurement  algorithms  devel¬ 
oped  for  use  with  this  study  and  the  hierarchical  equation  development. 
Chapter  Six  reviews  the  feasibility  work. 


Our  conclusions  and  recommendations  are  presented  In  Chapter  Seven. 
Appendices  Include  the  survey  forms  and  mathematical  modeling  equations. 


LITERATURE  RESEARCH 


2.1  SOURCES 

Before  constructing  precise  definitions  of  the  testability  measures 
of  fraction  of  faults  detected  (FFD) ,  fraction  of  faults  Isolated  (FFI), 
and  fraction  of  false  alarms  (FFA) ,  a  literature  survey  was  conducted  to 
gain  an  understanding  of  the  concepts  and  definitions  currently  In  circula¬ 
tion.  The  definitions  In  this  chapter  are  examples  of  published  defini¬ 
tions  and  are  not  reconmended  for  the  use  of  general  prediction  algorithms. 
The  recommended  definitions  appear  In  Chapter  Five  and  Appendix  E. 

There  Is  a  large  volume  of  literature  on  testability,  and  yet  there 
Is  little  consensus  on  the  definition  of  testability  terms,  because  of  the 
variety  of  intended  uses  for  the  literature  and  the  widely  varying  audi¬ 
ence.  The  literature  was  collected  from  numerous  sources  and  entered  Into 
a  bibliography.  Each  document  was  reviewed  to  find  definitions  of  or 
statements  concerning  FFD,  FFI,  and  FFA.  The  types  of  data  collected  and 
reviewed  Include  the  following: 

-  Military  Standards  and  Handbooks  (8) 

-  Reliability  and  maintainability  symposia  papers  (9) 

-  ARINC  Research  reports  (S) 

-  RADC  in-house  reports  (5) 

-  Other  contractor  reports  (42),  including  the  following:  Hughes 
Aircraft.  Lockheed.  IITRI,  Grumman.  ITT,  Sperry -Rand,  General 
Dynamics,  Boeing.  Westlnghouse,  Gould,  and  IDA 

In  addition,  abstracts  of  more  than  500  documents  were  reviewed. 

These  abstracts  were  provided  through  DTIC  literature  surveys. 


2.2  DEFINITIONS 

The  data  sources  were  used  to  obtain  definitions  or  concepts  of  FFD, 
PFI,  and  FFA.  The  definitions  were  varied.  About  20  percent  were  quanti¬ 
tatively  based;  the  other  80  percent  were  based  only  on  theory,  with  no 


operational  links.  The  following  subsections  give  examples  of  the  types 
of  definitions  found.  Detailed  descriptions  of  the  documents  cited  are 
presented  in  the  Bibliography. 

2.2.1  Fraction  of  Faults  Detected  (FFD) 

Fault  detection  is  the  capability  to  detect  and  indicate  one  or  more 
failures  within  the  equipment.  The  detection  and  indication  can  be  done 
by  BITE,  by  semiautomatic  means,  or  manually.  Fraction  of  faults  detected 
should  be  close  to  100  percent,  since  undetected  faults  can  be  hazardous 
to  a  mission  if  the  faulty  equipment  is  critical  to  the  mission.  In  the 
literature  surveyed,  there  were  more  definitions  for  "fault  detection" 
than  for  "fraction  of  faults  detected.”  The  following  paragraphs  give 
samples  of  the  definitions  found. 

The  RADC  report  Analytical  Procedures  for  Testability1  has  several 
related  definitions  of  FFD: 

-  Fraction  of  all  faults  detected  (or  detectable)  by  BIT/TE 

-  Fraction  of  all  detectable  faults  detected  (or  detectable)  with 
BIT/TE 

-  Fraction  of  all  faults  detected  through  use  of  defined  means 
("defined  means"  implies  all  means  of  detection  that  have  been 
identified) 

This  set  of  definitions  is  mostly  theoretical  and  not  quantitative. 
The  definitions  restate  the  same  concept  three  times.  The  difference  be¬ 
tween  the  first  two  is  that  the  second  one  clarifies  "faults”  to  "detect¬ 
able  faults,"  thus  assuming  that  there  are  undetectable  faults.  The  third 
definition  differs  only  in  its  reference  to  "how  detected." 

RADC  Testability  notebook  lists  five  definitions  accumulated  through 
surveys  for  "fraction  of  faults  detected": 

-  Percentage  of  all  faults  automatically  detected  by  BIT/ETE 

-  Percentage  of  all  faults  detectable  by  BIT/ETE 

-  Percentage  of  all  faults  detectable  on-line  by  BIT/ETE 

-  Percentage  of  all  faults  and  out-of-tolerance  conditions  detect¬ 
able  by  BIT/ETE 

-  Percentage  of  all  faults  detectable  by  any  means 
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This  set  of  definitions  is  also  more  theoretical  than  quantitative  and  is 
very  similar  In  context  to  the  first  set  of  definitions. 


There  were  two  literature  sources  that  stated  a  “fraction  of  faults 
detectable”  requirement.  General  Electric  Company  discusses  effectiveness 
of  fault  detection  as  the  number  of  failure  events  detected  correctly 
divided  by  the  number  of  actual  failures  experienced  and  states  that  the 

optional  effectiveness  range  should  be  between  85  percent  and  90  percent.1 

Sperry  Corporation  says  that  “fraction  of  faults  detected  is  a  SIT 
performance  requirement  for  not  less  than  98  percent  faults  detected  by 
the  operator  using  BIT.  BIT  shall  detect  failures  (and  out  of  tolerance) 
which  represent  at  least  90  percent  of  the  system  (or  subsystem)  probable 

failures."2 

The  Military  Standards,  including  the  new  MIL-STD-2165,  had  only 
definitions  of  fault  detection  and  no  quantitative  measures.  The  standard 
does  Include  fault  detection  as  an  element  of  system-level  test  effective¬ 
ness  In  BIT.  This  Is  given  as: 

l\  FD 

_  11 


where  Is  the  failure  rate  of  the  1^  item  and  FD^  Is  the  fault- 

detection  prediction  for  the  1th  item.  All  these  definitions  lack  spe¬ 
cifics.  It  Is  not  clear  where  to  draw  boundaries.  We  have  developed  a 
consistent,  mathematically  precise  definition  (Appendix  B)  that  will  be 
used  in  this  study. 

2.2.2  Fraction  of  Faults  Isolated  (FFI) 


Good  fault  Isolation  Is  the  ability  to  isolate  each  detected  fault 
quickly  and  accurately.  The  “fraction  of  faults  Isolated"  should  be  close 
to  100  percent  in  order  to  meet  the  mission-readiness  requirement.  Fault 
Isolation  can  be  accomplished  through  BITE,  semiautomatic,  or  manual 
fault-isolation  procedures.  Several  of  the  surveyed  sources  defined  or 
commented  on  fraction  of  faults  Isolated  or  related  concepts. 

Analytical  Procedures  for  Testability  defines  FFI  as  “the  fraction  of 


those  faults  detected  by  BIT/TE  which  are  then  isolated  with  BIT  to  the 
replacement  level  as  defined  by  the  maintenance  concept."  This  definition 
would  be  measurable  when  maintenance  reporting  cited  a  separate  code  for 
BIT/TE-triggered  maintenance  (often  a  separate  series  on  the  Job  control 
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mater)  and  tte  BIT  isolation  (sonatinas  In  comments,  but  not  always). 
The  algorithm  that  goas  with  this  definition  Is 


- -  ^FIbit/TE 

FFI  *  75 - 

iUBIT/TE 

where  FIBIT/TE  are  only  those  maintenance  actions  that  are  first  detected 
by  BIT/TE,  and  D^/ye  is  the  detection  of  malfunction  by  BIT/TE.  It  is 
not  clear  whether  or  not  this  should  include  false  alarms. 

The  summation  would  be  over  time  or  a  given  number  of  events. 

In  Assessment  of  Augmented  Electronic  Fuel  Controls  for  Modular  Engine 
Diagnostics  and  Condition  Monitoring.  GE  defines  FFI  as: 

m  number  of  failure  events  that  have  occurred 
number  of  maintenance  actions  to  correct 

This  equation  Is  similar  to  the  RADC  definition,  since  "number  of  failure 
events  that  have  occurred"  Is  equivalent  to  number  of  Isolated  faults,  and 
"number  of  maintenance  actions  to  correct"  is  equivalent  to  total  detected 
faults.  This  Is  also  a  measurable  definition. 

These  two  definitions  are  both  quantitative;  yet  there  are  certain 
unknowns  that  complicate  the  measurability  of  FFI.  Not  all  real  faults 
are  detected,  or  faults  are  detected  that  are  really  false  alarms.  He  may 
resolve  some  real  faults  as  "cannot  duplicate."  Thus  these  two  defini¬ 
tions  are  not  precise  enough  for  our  purposes,  and  a  more  Inclusive  equa¬ 
tion  will  be  derived  that  takes  Into  account  these  factors  affecting  faults 
Isolated. 

Military  specifications  such  as  MIL -STD-2165  or  MIL-STD-470A  provide 
requirements  on  fault-isolation  times  or  provide  general  definitions  such 
as:  "The  degree  to  which  a  test  program  or  procedure  can  Isolate  a  fault 
within  an  item;  generally  expressed  as  a  percent  of  the  cases  for  which 
the  Isolation  procedure  results  In  a  given  ambiguity  group  size"  (MIL- 
STD-2165) . 

2.2.3  Fraction  of  False  Alarms  (FFA) 

There  are  many  "fraction  of  false  alarms"  definitions.  Most  of  the 
inconsistencies  between  these  definitions  are  due  to  Inconsistencies  in 
the  definition  of  a  false  alarm. 

A  false  alarm  may  be  called  “an  indicated  fault  where  no  fault  exists” 
(MIL-STD-1309B) ,  or  a  "fault  Indication  of  a  failed  item  that  is  operating 
properly  instead  of  or  in  addition  to  designating  the  real  failure,"  or  a 
"failure  detection  that  cannot  be  repeated"  (our  survey;  see  Chapter 
Three).  These  three  definitions  are  not  consistent.  The  definitions  and 
comments  concerning  fraction  of  false  alarms  vary  also  depending  on  the 


Interpretation  of  a  false  alarm.  The  following  paragraphs  give  samples  of 
definitions  of  "fraction  of  false  alarms." 


The  RADC  report  Analytical  Procedures  for  Testability  lists  the  fol¬ 
lowing  three  definitions  of  “fraction  of  false  alarms": 

-  Fraction  of  all  BIT/TE-indlcated  faults  which  are  false  alarms 

-  Ratio  of  quantity  of  BIT/TE  false  alarms  to  quantity  of  faults 
detected  through  use  of  defined  means 

-  Ratio  of  false  alarms  to  actual  faults 

These  may  seem  to  be  quantitative  definitions  but,  unless  there  Is  a  clear 
measure  of  a  false  alarm,  the  quantity  of  false  alarms  Is  not  measurable 
either. 

Another  FFA  definition  is  from  BIT/External  Test  Figures  of  Merit  and 

Demonstration  Techniques1:  “Fraction  of  false  alarms  is  the  fraction  of 
all  BIT/TE-lndlcated  faults  which  are  false  alarms.  False  alarms  are 
those  indications  of  a  fault  when  an  actual  fault  has  not  occurred."  This 
definition  Is  better  in  that  It  defines  Its  interpretation  of  a  false 
alarm. 

MIL-STD-2165  defines  a  false  alarm  as  a  fault  Indicated  by  BIT  or 
other  monitoring  circuitry  where  no  fault  exists. 

RADC  Testability  Notebook  defines  a  false  alarm  as  an  Indicated  fault 
where  no  fault  exists  (does  not  Include  good  Items  In  an  ambiguity  group) . 
This  latter  definition  Is  In  agreement  with  MIL-STD-I309B.  This  publica¬ 
tion  lists  the  following  measures  of  effectiveness  related  to  “fraction  of 
false  alarms": 


-  Rate  at  which  false  Indications  occur  (per  106  hours) 

-  Percentage  of  Indicated  failures  caused  by  actual  failures 

-  Percentage  of  BIT/ETE-indicated  failures  caused  by  actual  failures 

-  Percentage  of  BIT/ETE  fault  isolations  to  the  wrong  UUT 

These  measures  are  theoretically  based  and  suffer  some  measurability 
problems.  The  first  one  does  not  define  a  false  Indication.  The  next  two 
avoid  using  “false  Indications"  and  use  only  "actual  failures"  Instead  as 
a  way  around  it.  The  last  one  falls  to  Indicate  how  to  measure  a  fault 
Isolation  to  a  wrong  unit  under  test.  The  last  three  can  be  algorithmi¬ 
cally  developed  but  lack  the  precision  necessary  for  this  study. 


xHughes  Aircraft  Company,  RADC-TR-79-309 ,  December  1979. 


2.3  COMMENTS  ON  DEFINITIONS 

The  wide  variety  of  definitions  available  provides  a  somewhat  con¬ 
fusing  array  of  possibilities  that  are  not  totally  consistent.  Many  are 
so  tailored  to  the  measurement  of  specific  details  that  they  have  limited 
use;  others  provide  only  a  theoretically  based  descriptor  for  discussion 
purposes.  Those  which  are  quantitative  are  unmeasurable  at  the  organiza¬ 
tional  level. 

In  examining  almost  500  related  documents  (Including  military 
standards/handbooks .  symposium  proceedings,  and  manufacturer  and  contrac¬ 
tor  reports) ,  we  encountered  an  almost  endless  variety  of  bookkeeping 
algorithms.  In  Itself  this  variety  Is  not  bad.  because  many  of  these 
documents  are  directed  toward  specific  hardware  or  analysis  problems  and 
the  definitions  are  somewhat  tailored.  It  does  make  It  difficult  to  keep 
definitions  compatible  with  most  of  the  current  literature.  Perhaps  the 
single  largest  shortcoming  in  the  definitions  discovered  In  our  literature 
search  was  the  lack  of  a  consistent  set  of  definitions  for  FFD.  FFI,  and 
FFA.  While  several  of  the  Individual  definitions  are  usable,  no  matched 
set  exists.  Our  definitions  are  based  on  the  relational  properties  of  a 
good  MgenerlcN  definition,  that  is,  a  definition  not  Intended  for  use  In 
solving  a  specific  hardware  or  analysis  problem.  The  relational  properties 
are  as  follows: 

-  The  definitions  should  be  in  accordance  with  military  standards 
and  handbooks. 

-  The  definitions  should  be  consistent  with  each  other. 

-  The  definitions  must  be: 

—  Consistent  with  the  intuitive  Interpretation  of  the  parameter 
being  defined 

—  Directly  or  indirectly  related  to  mission  readiness  factors 
—  Mathematically  precise 
—  Measurable 

—  At  least  experimentally 
—  Possibly  by  specialized  field  reporting 
—  Possibly  through  modification  of  standard  field  reporting 
—  Capable  of  being  specified 
—  Capable  of  being  demonstrated 

-  The  defined  quantities  should  be  predictable. 
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None  of  the  definitions  we  examined  met  all  of  these  requirements, 
m  fact,  the  preceding  requirements  probably  fora  an  overspecification. 

The  definitions  derived  for  this  study  and  presented  in  Appendix  B  are 
primarily  based  on  the  set  theory  model  of  the  organizational  maintenance 
system  and  result  from  compromises  and  iterations  with  all  of  the  models. 
They  meet  the  properties  of  precision  and  measurability  and  are  totally 
consistent,  but  they  may  or  may  not  meet  the  other  requirements.  Of  par¬ 
ticular  Interest  is  the  hierarchical  relationship  between  the  system  defi¬ 
nitions  and  the  subsystem  definitions,  as  discussed  in  detail  in  Chapter 
Five. 


MAINTENANCE  SURVEY 


3.1  BASIS  OF  SURVEY 

Chapter  Two  described  the  wide  variety  of  research  that  has  been  con¬ 
ducted  in  the  testability  field.  It  was  found  that  there  is  little  or  no 
consensus  regarding  the  definition  of  testability  terms.  While  this 
appears  to  offer  a  wide  latitude  in  the  establishment  of  a  set  of  defini¬ 
tions  for  developing  predictor  procedures,  it  was  important  to  stay  within 
the  bounds  of  intuitive  reasoning.  At  the  same  time,  it  was  necessary  to 
locate  sources  of  enriched  data  to  supplement  the  mass  of  data  to  be 
analyzed.  Finally,  it  was  desirable  to  take  advantage  of  the  years  of 
hands-on  experience  within  the  military  system  to  provide  insights  into 
the  maintenance  and  reporting  systems.  The  development  of  a  maintenance 
survey  was  thought  to  be  the  most  prudent  approach  to  satisfying  these 
requirements. 


3.2  ORGANIZATIONS  SURVEYED 

A  total  of  108  organizational-level  maintenance  centers  were  sur¬ 
veyed.  including  the  Strategic  Air  Command  (SAC),  the  Tactical  Air  Command 
(TAG),  and  the  Military  Airlift  Comnand  (MAC),  as  well  as  U.S.  Navy  Air 
Wings.  Air  National  Guard  (ANG),  and  commercial  aviation  groups.  The  last 
three  categories  were  surveyed  for  completeness,  with  concentration  on  the 
Air  Force  groups  for  data  assistance  and  model -building  efforts.  Follow¬ 
up  visits  were  made  to  at  least  one  MAC  and  SAC  operational  unit,  and  sev¬ 
eral  visits  were  made  to  TAC  units.  Most  of  the  commands  surveyed  and 
interviewed  had  both  organizational- level  and  intermediate-level  mainte¬ 
nance.  The  visits  clarified  survey  responses  and  provided  a  user-level 
view  of  the  modeling  efforts  described  in  Appendix  B.  All  visited  commands 
provided  assistance  in  structuring  the  flow  model. 


3.3  SURVEY  CONTENTS 

Project  team  members  drew  up  initial  survey  questions  to  be  con¬ 
sidered.  The  questions  were  distributed  to  RADC  and  throughout  ARINC 


Research  for  formal  review  and  comment.  In  addition,  several  of  our  cur¬ 
rent  clients  were  approached  for  comments  on  an  informal  basis.  The 
resulting  survey  form  Is  presented  in  Appendix  A. 

The  survey  questions  centered  on  five  major  areas: 

-  Number  of  systems  maintained  and  who  maintains  them 

-  Reporting  systems  that  Information  Is  received  from  and  sent  to 

-  Local  files  maintained  and  access  to  those  files  (this  Information 
was  sought  for  data  enrichment) 

-  Intuitive  definitions  of  detection.  Isolation,  and  false  alarms 

-  Philosophy  and  Insights 

To  help  maximize  survey  response,  (1)  the  respondents  were  assured 
anonymity,  and  (2)  the  survey  form  was  limited  In  length.  The  first  factor 
helped  in  obtaining  candid  answers,  and  the  second  served  to  minimize  the 
effort  involved  in  filling  out  the  form. 


3.4  SURVEY  RESPONSE 

Although  some  of  the  groups  contacted  failed  to  respond  to  official 
survey  Inquiries,  a  substantial  return  rate  was  achieved.  Response  sta¬ 
tistics  are  provided  In  Table  1. 


TABLE  1 .  SUMMARY  OF  SURVEY  RESPONSES 


Item 

Quantity 

Percent 

Organizational-level  main¬ 

108 

100 

tenance  centers  surveyed 

Responses  received  to  date 

47 

44 

3.5  SURVEY  ANALYSIS 

Table  2  provides  summary  data  for  the  surveys  returned.  A  total  of 
27  units  maintained  local  data  files  in  addition  or  as  a  supplement  to 
the  standard  reporting  systems.  Of  these,  15  were  both  available  for 
research  and  met  the  requisite  data  criteria,  thus-  adding  to  our  data 
resources.  Of  particular  interest  were  multiple-failure  resolution  and 
the  “bad  actors-  files.  Almost  all  of  the  units  surveyed  kept  track  of 


“bad  actors,"  but  they  did  not  all  have  local  data  bases;  those  that  did 
not  relied  heavily  on  personnel  experience  and  expertise.  In  addition, 
multiple  related  failures  were  generally  handled  by  separate  reporting  of 
the  Individual  malfunctions.  Sometimes  comments  on  AFTO  349  can  confirm 
some  correlation  of  failures,  but  these  cannot  consistently  be  defined  on 
the  basis  of  the  standard  reporting. 

While  most  of  the  Information  was  extremely  useful,  care  should  be 
taken  In  using  Table  2.  because  some  of  the  responses  to  the  survey  can  be 
somewhat  misleading.  For  example,  the  respondents  overwhelmingly  affirmed 
differentiation  between  operator  complaints  and  normal  system  maintenance. 
Closer  examination  of  the  responses  shows  that  these  complaints  are 
reported  by  when-discovered  codes,  job  control  number  prefixes,  or  com¬ 
ments  provided  on  AFTO  349.  There  are  basic  holes  In  this  Information 
flow.  Pilot  reports  of  malfunctions  indicated  by  use  of  preflight  check 
lists  are  categorized  as  operator  complaints.  Often,  failures  that  should 
result  In  BIT-generated  job  control  numbers  appear  only  when  the  pilot  did 
not  report  these  failures  on  debrief.  These  factors  are  discussed  In  more 
detail  In  Chapters  Four  and  Six. 

Of  primary  Interest  are  the  responses  to  the  Intuitive  questions  that 
are  the  last  three  entries  In  Table  2.  These  questions  concern  the  Intui¬ 
tive  definition  of  false  alarm,  faults  detected,  and  faults  Isolated.  A 
general  consensus  on  Intuitive  definitions  would  be  expected,  but  more 
than  a  third  of  the  responses  were  other  than  what  had  been  predicted.  In 
addition,  while  opinions  were  strongly  expressed  (51  written  responses), 
there  was  no  consensus.  In  fact,  no  one  response  to  any  of  the  questions 
approached  50  percent.  This  result  is  consistent  with  the  literature 
search  and  points  out  the  need  for  mathematically  precise  and  consistent 
definitions. 

In  addition  to  the  survey  responses.  15  local  data  bases  located  at 
the  organizational  level  are  available  for  further  study. 


3.6  SUMMARY  AND  CONCLUSIONS  OF  SURVEY 

The  survey  filled  three  basic  needs  of  the  research  project: 

-  We  were  able  to  locate  and  identify  data  sources  for  enriching  and 
supplementing  the  normal  Air  Force  Maintenance  Data  Collection 
System. 

-  We  were  provided  with  an  introduction  to  several  of  the  commands 
and  reviewed  their  maintenance  procedures  and  our  models  of  orga¬ 
nizational  maintenance.  (This  worked  out  very  well  in  that  the 

'  development  of  the  flow  model  would  have  been  Impossible  without 


the  participation  of  the  commands.  The  model  ultimately  became 
recognized  by  organizational  maintenance  personnel  as  an  accurate 
representation  of  organizational  maintenance  in  the  Air  Force.) 

We  found  a  diversity  of  "intuitive"  concepts  of  faults  detected, 
faults  Isolated,  and  false  alarms,  which  confirmed  our  literature 
surveys  and  reaffirmed  the  need  for  precise,  quantifiable,  and 
measurable  organizational-level  measures. 


CHAPTER  POUR 


DATA  SOURCES 


A  wide  variety  of  data  resources  were  either  utilized  during  the 
Phase  I  study  or  developed  for  the  Phase  II  study.  These  resources  cover 
the  depth  and  breadth  of  maintenance  reporting  and  testing. 

We  have  identified  the  following  three  levels  of  data  as  being  neces¬ 
sary  for  the  development  of  testability  predictors: 

-  Field  Data  -  Data  collected  through  routine  reporting  channels 

-  Engineering  Test  Data  -  Data  from  validated  field  tests  of  limited 
duration  and  observed  by  an  engineer 

-  Design  Data  -  Detailed  engineering  analysis  of  design-related 
parameters 

During  the  course  of  our  Invest lgat Ion ,  we  were  able  to  assemble  the 
data  summarized  In  Table  3.  To  minimize  the  risk  Involved  In  predictor 
development,  all  three  levels  of  data  should  be  used.  Each  of  the  three 
levels  are  discussed  in  the  following  subsections.  The  data  appropriate 
to  these  studies  were  derived  from  a  much  larger  list  of  candidate  data 
resources. 


4.1  FIELD  DATA 

The  primary  source  of  field  data  to  support  this  study  Is  the  AFR  66-1 
promulgated  Maintenance  Data  Collection  System  (MDCS).  It  is  discussed  In 
detail  In  Air  Force  Technical  Order  00-20-02:  we  will  address  those  aspects 
of  MDCS  which  are  particularly  germane  to  this  study. 

4.1.1  A FTP  Form  349 

The  basic  source  of  data  for  the  MDCS  Is  AFTO  Form  349,  shown  In 
Figure  1.  Every  reported  maintenance  action  performed  on  aircraft  avionics 
generates  an  AFTO  349.  There  are  a  number  of  Items  on  the  form  that  are 
particularly  Important  to  this  study. 

The  first  important  entry  Is  the  job  control  number  (JCN)  (block  1). 
Every  maintenance  action  Is  assigned  a  unique  JCN,  which  stays  open  until 
the  action  is  complete.  Thus,  one  JCN  equals  one  maintenance  action. 
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Since  the  first  three  digits  of  the  JCN  code  denote  the  Julian  day.  cer¬ 
tain  rate  information  Is  available  as  well. 

Block  3  of  AFTO  349  Identifies  the  equipment  on  which  work  was  per¬ 
formed  or  from  which  an  Item  was  removed.  The  block  allows  sorting  of 
maintenance  actions  by  major  equipment  type. 

The  work  unit  code  (block  C)  further  refines  the  Identity  of  the  Item 
being  serviced.  Each  avionic  LRU  (and  SRU)  for  each  aircraft  type  has  a 
unique  work  unit  code,  which  allows  a  sorting  of  data  by  LRU  type.  That 
Is,  one  could  construct  a  data  base  of  AFTO  349  data  and  extract  all  main¬ 
tenance  actions  on  a  given  LRU  for  a  given  aircraft. 

The  action-taken  code  (block  B)  Indicates  the  action  or  actions  taken 
to  resolve  the  original  discrepancy,  such  as  removal  and  replacement  of  an 
SRU. 

The  when-discovered  code  (block  E)  Indicates  the  point  In  a  mission 
when  the  discrepancy  being  corrected  was  discovered.  Typical  entries  are 
pre-flight  (no  abort),  pre-flight  (abort),  in-flight  (no  abort),  post- 
flight,  and  special  Inspection. 

The  how-malfunctioned  code  (block  F)  Is  a  three-digit  code  that 
describes  the  malfunction  In  general  terms.  Acceptable  codes,  which  are 
listed  In  the  maintenance  job  guides  for  the  major  equipment  types  (the 
-06  series  Technical  Orders),  may  be  slightly  different  for  each  major 
equipment.  That  is.  "how  mal"  codes  for  the  C-5  may  be  different  from 
those  for  the  F-15.  Certain  codes  are  standard.  Including  "cannot  dupli¬ 
cate"  (CND) ,  i.e..  Code  799.  This  standardization  provides  a  method  for 
estimating  one  of  the  testability  parameters,  as  will  be  seen  In  Chapters 
Five  and  Six. 

4.1.2  Assessment  of  MDCS  Data  Sources 

There  Is  a  major  shortcoming  of  the  MDCS  when  it  Is  used  for  test¬ 
ability  analysis.  This  should  not  be  viewed  as  an  Indictment  of  the  MDCS; 
the  system  appears  to  do  a  good  job  when  it  is  used  for  its  original  pur¬ 
pose.  It  does  provide  a  measure  of  maintenance  productivity,  equipment 
reliability,  and  maintenance  and  support  costs.  However,  the  thrust  of 
AFTO  349  is  toward  maintainability  (what  failed,  how  long  it  took  to 
repair,  and  so  on).  The  central  Issue  in  testability  is  how  effective  the 
maintenance  system  (built-in-test  software,  check  lists)  is  in  discovering 
and  isolating  faults.  As  can  be  seen  from  the  discussion  in  Chapter  One. 
and  as  will  be  amplified  In  Chapters  Five  and  Six,  a  critical  question 
neither  asked  on  AFTO  349  nor  uniformly  documented  anywhere  is  whether  the 
fault  was  detected  or  Isolated  by  the  "normal  system"  maintenance. 

"Normal  system"  here  is  the  set  of  maintenance  aids  provided  as  part  of 
the  entire  system  —  test  gear  check  lists,  built-in-test  software,  and 
other  aids.  A  number  of  responses  to  the  maintenance  survey  (see  Chapter 
Three)  indicated  that  information  of  this  type  is  retained.  However,  it 
is  not  uniformly  coded  on  AFTO  349;  hence,  it  is  very  difficult  to  conduct 
reasonably  accurate  statistical  surveys  dealing  with  this  parameter. 


A  second  flaw  In  the  use  of  (CCS  for  testability  analysis  Is  that 
certain  responses  on  AFTO  349s  nay  be  politically  damaging  and  thus  tend 
to  be  avoided.  For  example,  "cannot  duplicate"  (CM))  might  be  Interpreted 
as  resulting  from  poor  training  or  poor  fault-isolation  procedures.  Thus, 
pressures  might  be  placed  upon  maintenance  personnel  to  avoid  use  of  CM) 
for  "how  mal."  We  will  touch  on  this  Issue  In  Chapter  six. 

Nonetheless,  the  Maintenance  Data  Collection  System  Is  the  only 
source  of  Air  Force-wide  automated  data  analysis  available.  Any  further 
study  must  recognize  and  deal  with  the  above  shortcomings. 

4.1.3  Specific  MDCS  Sources 

Table  3  lists  a  number  of  sources  of  field  maintenance  data.  Most 
fruitful  for  this  Phase  I  study  were  the  F-15  avionics  AFTO  349  summaries 
provided  by  the  1st  TAC  Fighter  Wlng/MA.  and  the  F-16  enhanced  ST/BIT 
reports.  The  latter  Included  a  series  of  pilot  debrief  reports  that  could 
be  correlated  to  AFTO  349s  by  matching  JCNs.  Unfortunately,  this  proved 
to  be  such  a  time-consuming  manual  task  that  we  were  only  able  to  effec¬ 
tively  use  data  from  two  of  the  four  bases.  In  addition,  the  debrief 
reports  could  not  be  used  to  reliably  partition  maintenance  actions  Into 
"normal"  and  "other  than  normal"  system  maintenance  as  was  hoped.  The 
absence  of  a  set  of  BIT  codes  did  not  mean  that  BIT  did  not  detect  a  fault, 
nor  did  It  mean  that  the  pilot  had. 

A  third  source  of  data,  and  one  that  we  believe  holds  the  greatest 
promise  for  useful  analysis.  Is  the  C-5  Malfunction  Analysis  Detection  and 
Recording  System  (MADARS).  Unfortunately,  there  was  a  complete  mismatch 
between  the  field  data  (MADARS)  and  engineering/design  data  (such  as  Tech¬ 
nical  Orders)  available  to  us.  This  will  be  corrected  during  Phase  II  If 
we  utilize  the  MADARS  data.  This  system  Is  attractive,  because  It  auto¬ 
mates  the  AFTO  349  reporting  system  in  a  fairly  discrete  set.  There  are 
three  C-5  bases  —  Tinker  AFB,  Oklahoma;  Travis  AFB,  California;  and  Dover 
AFB.  Delaware  —  which  raises  the  possibility  of  more  easily  implementing 
an  expanded  data-gatherlng  system. 

Two  larger  management  Information  systems  are  listed  In  Table  3  — 
the  Weapon  System  Management  Information  System  (WSMIS)  and  the  Mainte¬ 
nance  and  Operational  Data  Analysis  System  (MODAS).  Both  obtain  their 
source  data  from  AFTO  349s  and  thus  represent  no  new  Information.  They 
may.  however,  prove  fruitful  for  developing  estimations  for  certain  por¬ 
tions  of  the  key  testability  parameters  as  a  result  of  their  scope.  They 
will  be  further  assessed  in  Phase  II. 


Finally,  approximately  15  other  local  data  bases  were  Identified 
as  a  result  of  the  survey  discussed  In  Chapter  Three.  These  were  not 
Investigated. 


4.2  ENGINEERING  TEST  DATA 


This  level  of  data  was  the  sparsest  of  the  three  levels  utilized  In 
Phase  I.  Development  of  these  data  will  be  necessary  in  Phase  II  of  this 
study  and  will  represent  the  MparameterlzlngH  data  necessary  to  build 
predictions  successfully. 


4.3  DESIGN  DATA 

There  was  a  mismatch  between  the  level  of  detail  available  in  design 
data  and  that  available  In  field  data,  He  had  a  large  amount  of  field 
data  on  F-16  LRUs,  but  we  did  not  have  complete  access  to  F-16  LRU  design 
data.  Through  the  help  of  the  F-16  SPO,  however,  we  were  able  to  Identify 
certain  design-related  data  from  intermediate  maintenance  Technical 
Orders.  If  F-16  field  data  become  key  to  Phase  II,  we  will  take  action  to 
obtain  intermediate-level  and  depot-level  maintenance  manuals  for  the  LRUs 
In  question. 

The  situation  was  similar  for  the  F-IS:  there  were  two  F-15  LRUs 
(the  Inertial  Navigation  Unit  and  the  Low  Band  RF  Amplifier)  for  which  we 
were  unable  to  obtain  technical  design  data.  Fortunately,  we  had  avail¬ 
able  the  current  Intermediate-level  maintenance  manual  for  the  three 

APG-63  LRUs  shown  in  Table  3.  There  is  a  STAMP1  testability  analysis 
under  way  on  the  F-15  APG-63  Radar:  results  of  the  analysis  will  be  avail¬ 
able  for  evaluation  with  field  data  during  Phase  II. 

He  currently  have  no  technical  design  data  on  the  C-5  avionics  sys¬ 
tems;  this  will  be  a  priority  item  for  Phase  II  if  the  C-5  Is  chosen  for 
predictor  development. 

If  sufficient  field  data  cannot  be  obtained,  we  have  detailed  STAMP 
analyses  of  13  systems,  as  shown  In  Figure  2.  The  majority  of  these  sys¬ 
tems  are  EH  mission  avionics.  If  we  must  employ  these  analyses  in  lieu  of 
field  data,  there  may  not  be  enough  data  points  in  an  equipment  spectrum 
that  is  broad  enough  to  provide  high-confidence  estimators.  For  that  rea¬ 
son.  the  STAMP  studies  are  viewed  as  a  "method  of  last  resort." 


1 STAMP  (System  Testability  and  Maintenance  Program)  is  a  detailed  test¬ 
ability  model  developed  by  ARINC  Research  for  design  testability  analysis 
and  fault-isolation  strategy  development.  It  has  been,  and  is  being, 
applied  to  a  number  of  military  systems. 


Project  Attributes 


Project  Name 


Goodyear  Atomic 
(Various  Gas  Centrifuge 
Enrichment  Systems) 

• 

• 

• 

• 

ALQ-131  (EW) 

• 

• 

R-SASE  (EW  ATE) 

• 

• 

• 

• 

UH-60  Stability 
Augmentation  System 

• 

• 

• 

CARA 

• 

• 

• 

• 

• 

EA-6B  Exciter  (EW) 

• 

• 

• 

A-6E  Detection  and 
Ranging  Set 

• 

• 

• 

ALQ-184  (EW) 

• 

• 

• 

• 

Advanced  Avionics 

System 

• 

• 

• 

• 

• 

• 

APG-63  Radar 

• 

• 

• 

• 

MSQ-103  (ELINT) 

• 

• 

• 

• 

• 

• 

ALR-67  (EW) 

• 

• 

• 

• 

• 

ALR-62  (EW) 

• 

• 

• 

• 

FIGURE  2.  AVAILABLE  STAMP  ANALYSES 


4.4  SUJMARY  OF  DATA  SOURCES 


During  this  Phase  I  effort,  we  learned  that  the  Maintenance  Data  Col¬ 
lection  System  may  not  be  adequate  in  its  current  form  for  developing  pre¬ 
dictors  of  FFA,  FFI,  and  FAR.  A  revised  or  enhanced  collection  system 
must  be  structured  to  provide  the  appropriate  data.  Since  the  c-5  MADARS 
is  fairly  small  (compared  to  the  MDCS)  and  automated,  it  may  be  the  best 
candidate  for  such  a  restructuring.  We  will  need  to  gather  detailed 
technical/design  data  on  the  C-5  avionics  systems  if  such  a  restructuring 
is  undertaken. 


CHAPTER  FIVE 


APPLICATION  OF  ALGORITHMS 


5.1  MATHEMATICAL  MODELING 

Mathematical  modeling  was  undertaken  to  provide  measurement  algo¬ 
rithms  that  were  consistent  with  the  derived  definitions.  We  started  with 
the  modified  state  model,  which  Is  intended  to  relate  the  organizational 
testability  parameters  to  a  search  for  the  system  state.  This  represents 
the  heart  of  the  maintenance  problem  In  that  there  Is  an  indication  of  a 
problem  and  action  must  be  taken  to  find  out  whether  a  real  failure  is 
present  and  where  It  Is  located.  As  this  model  was  being  developed,  it 
became  apparent  that  conflicts  In  definitions  were  surfacing.  For  exam¬ 
ple.  It  was  not  clear  whether  a  “cannot  duplicate"  (CND)  event  and  a 
fault-isolation  event  were  mutually  exclusive.  To  resolve  these  problems, 
a  second  model,  based  on  membership  In  sets,  was  developed.  The  primary 
tool  was  the  Venn  diagram.  In  which  both  mutual  exclusivity  and  coincident 
properties  are  explicit.  This  set  model  led  to  a  clear  and  concise  set  of 
definitions  that  were  mathematically  precise,  as  well  as  an  algorithm  set 
that  could  be  used  to  verify  the  other  models.  The  state  model  was  then 
reworked  on  the  basis  of  definitions  generated  by  the  set  theory  model 
with  most  of  the  conflicts  resolved. 

A  third  model,  based  on  the  flow  of  maintenance  events,  was  developed 
and  was  pursued  concurrently  with  the  other  two  models.  This  model  was  to 
solve  two  of  the  problems  being  faced.  The  first  problem  was  relating 
maintenance  actions  to  readiness.  While  a  preliminary  connective  had  been 
established  with  the  modified  state  model.  It  was  less  than  satisfactory. 
The  flow  model  would,  by  tracing  events  through  the  mission/maintenance 
cycle,  provide  a  direct  tie-in.  The  second  problem  was  more  basic:  There 
was  no  direct  way  to  relate  what  we  had  done  mathematically  to  the  mainte¬ 
nance  personnel.  The  first  two  models  were  too  “mathematical."  The  flow 
model  was  readily  analyzed  by  maintenance  personnel  of  SAC,  TAC,  and  MAC, 
and  underwent  major  revisions  based  on  discussions  with  these  personnel. 

As  a  clearer  picture  of  the  organizational-level  maintenance  process 
evolved,  modifications  were  made  to  both  of  the  other  models.  Finally,  a 
flow  model  evolved  that  was  satisfactory  to  both  organizational-level 
maintenance  personnel  and  the  mathematicians. 

A  detailed  review  of  each  of  the  models  Is  presented  in  Appendix  B. 
The  final  form  of  the  definitions  is  reviewed  here  and  is  discussed  in 
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both  Appendix  B  and  the  Executive  Summary.  It  Is  the  purpose  of  this  chap¬ 
ter  to  relate  the  measurement  models  of  Appendix  B  to  the  problem  of  com¬ 
puting  actual  values  and  to  build  up  system/subsystem  algorithms.  As  will 
be  seen  In  Chapter  Six,  the  state  model  most  closely  approaches  the  current 
measurement  systems.  Algorithmic  representations  will  be  drawn  by  use  of 
this  model,  although  the  same  systematic  procedures  can  be  used  (although 
the  specific  algorithms  will  change)  for  any  of  the  models.  This  chapter 
Is  broken  down  according  to  the  principal  terms  FFD,  FFI,  and  FFA,  with  a 
breakdown  by  system,  subsystem,  and  system/subsystem  relationship. 

The  following  paragraphs  describe  some  basic  Items  that  apply  to  the 
mathematics  throughout  this  chapter.  The  subscript  "s"  refers  to  normal 
system  maintenance  (NSM) .  We  define  NSM  as  follows: 

Techniques  that  are  specified  as  standard  operating 
procedures  for  use  of  BIT,  ATE.  semiautomatic,  or  doc¬ 
umented  manual  detection  and  troubleshooting  for  a 
given  system  under  test.  They  Include  regular  calen¬ 
dar  checks  and  normal  "goH  checks.  NSM  is  sometime 
called  “defined  means." 

Thus  £MA  refers  to  maintenance  actions  triggered  by  the  NSM,  and 
s 

£MA  Is  all  maintenance  actions.  Unless  otherwise  specified,  a  l  sym¬ 
bol  refers  to  events  over  time,  so  that  £MA  Is  the  sum  of  all  mainte¬ 
nance  actions  on  the  system  over  some  measurement  period. 

The  subscript  “ss“  refers  to  a  subsystem.  The  system/ subsystem  bound¬ 
ary  Is  an  artificial  one,  and  Is  drawn  on  the  basis  of  analysis  needs.  A 
system  Is  taken  as  a  functional  or  structural  entity.  Its  boundaries  are 
often  physical  break  points  between  the  system  and  surrounding  systems. 
Examples  are  an  entire  aircraft,  or  a  weapon  system,  or  even  a  single  LRU 
within  a  weapon  system.  A  subsystem,  on  the  other  hand.  Is  any  portion 
less  than  the  totality  of  the  universe  of  concerns.  For  example,  the 
Government,  concerned  with  the  acquisition  of  Weapon  System  A,  would  con¬ 
sider  LRU  5  a  subsystem  of  Weapon  System  A.  However,  the  contractor  who 
builds  LRU  5.  and  only  LRU  5.  In  Weapon  System  A  would  consider  LRU  5  a 
system.  The  subscript  "ssc"  refers  to  a  subsystem  contribution  as  In  the 
LRU  5  contribution  to  Weapon  System  A.  Finally,  an  "1"  subscript  will  be 

used  to  refer  to  the  1th  subsystem  In  a  system. 


5.2  FRACTION  OF  FAULTS  DETECTED  (FFD) 

Fraction  of  faults  detected  should  ideally  be  unity.  It  is  the  entry 
point  for  maintenance.  We  will  discuss  a  system- level  value  of  FFD  fol¬ 
lowed  by  subsystem  values  of  FFD,  and,  finally,  their  interrelationships 
with  each  other. 


The  set  theory  system  definition  derived  for  fault  detection  as  taken 
from  Appendix  B  is: 

Fault  Detection  -  Normal  system  maintenance  indicates 
that  the  system  is  not  functioning  properly,  and  this 
Indication  is  the  result  of  a  real  fault  within  the 
system. 

To  relate  this  to  FFD,  we  must  normalize  by  the  faults  within  the  sys¬ 
tem.  Note  that  we  are  dealing  only  with  relevant  failures  as  discussed  in 
Chapter  One  and  Appendix  B.  The  fraction  of  faults  detected  at  the  system 
level  will  then  be  given  by:  "the  ratio  of  fault  detection  to  faults  in 
the  system."  From  the  Appendix  B  modified  state  representation,  this 
translates  algorithmically  to 

/IMRS  *  0sKNDs\ 

FFD  -  I  — -  --  nVrMn  /  (Equation  54  of  Appendix  B) 


The  numerator  of  this  term  represents  the  fault  detection  (that  is,  the 
NSM-triggered  maintenance  actions  minus  the  false  alarms  generated  by  NSM) . 
The  denominator  represents  the  faults  in  the  system  (that  is,  the  total 
maintenance  actions  minus  all  false  alarms). 

The  system- level  fault  detection  can  be  measured  by  knowing  the  NSM- 

generated  maintenance  actions  (MA  ) .  the  "cannot  duplicate"  results  of 

s 

NSM-generated  maintenance  actions  (CNDs>.  the  total  maintenance  actions 

(flA),  and  the  total  "cannot  duplicate"  ( CND )  events,  all  of  which  may  be 

measurable  at  the  organizational  maintenance  level.  The  term  0  £CND 

s  s 

represents  the  false  alarms  due  to  NSM.  The  term  3ICND  represents  all 

false  alarms.  The  factors  are  derived  in  Appendix  B.  3  and  3  are 

s 

empirical  coefficients  and  represent  the  percentage  of  CNDs  that  are  false 
alarms.  These  will  be  empirically  determined  during  the  Phase  II  work  and 
presented  in  tabular,  graphical,  or  functional  forms. 

5.2.2  FFD  -  Subsystem 

The  subsystem  definitions  are  based  on  a  participating  element  of  a 
system.  If  the  subsystem  is  all  that  is  under  consideration  (that  is.  a 
system  boundary  is  drawn  around  the  subsystem)  then  the  system-level  defi¬ 
nitions  apply  to  that  subsystem.  For  a  subsystem,  the  set  theory-derived 
definition  for  fault  detection  is: 


Fault  Detection  -  NSM  indicates  that  a  subsystem  is 
not  functioning  properly,  because  of  a  real  fault 
within  the  system.  The  detection  can  be  proper  or 
Improper . 


Proper  Detection  -  Fault  Is  within  the  subsystem  in  which  detection 
occurs . 

Improper  Detection  -  Fault  is  within  a  subsystem  other  than  the  one 
in  which  the  detection  occurs. 

To  relate  this  to  FFD,  we  must  normalize  by  the  faults  within  either 
the  system  or  the  subsystem.  The  fraction  of  faults  detected  at  the  sub¬ 
system  level  will  then  be  given  by: 

The  ratio  of  subsystem- level  detections  to  either 
faults  within  the  system  or  faults  within  the  sub¬ 
system.  The  subsys tern- level  contribution  will  be  the 
ratio  of  the  sum  of  all  detections  to  the  sum  of  all 
faults  within  the  system.  The  subsystem  FFD  will  be 
the  ratio  of  the  sum  of  the  proper  subsystem  detec¬ 
tions  to  the  siim  of  all  faults  within  the  subsystem. 

From  the  Appendix  B  modified  state  representation  these  definitions 
translate  algorithmically  to: 

Subsystem  contribution: 


(X«As  -  3sXCNDs), 


(IMA  -  PlCND)  _ 


subsystem  detections 


subsystem  faults 


This  equation  requires  the  same  data  base  as  the  system- level  FFD,  except 
that  we  must  now  partition  the  data  on  the  basis  of  subsystem  properties: 


Subsystem  FFD: 


(IMA  -  3  ICND  ) 

s  s  s  proper,  ss 

(IMA  -  3ICND) 


proper  subsystem  detections 
subsystem  faults 


This  equation  shows  that  if  the  subsystem  is  considered  a  system,  an 
Improper  detection  may  become  a  CND. 

5.2.3  FFD  Svstem/Subsystem  Relationshi 


The  definitions  of  system  and  subsystem  FFD  can  be  related  by  Equa 
tlons  40  to  42  of  Appendix  B  as: 

^^ss.  ^ss,  improper 


where  IFD  represents  subsystem  faults  detected  [also  represented  by 

SS 

(IMA  -  3  ICND  )]  and  IF  represents  system  faults  [also  represented 


by  (JMA  -  g£CND)]  [an  improper  subsystem  fault  detection  (FD  ,  improper) 

ss 

is  as  defined  in  Section  5. 2. 2], or 

FFD  =  l  (T4FFD  ) 

^ss  i  ssc 

where  Y^  allocates  the  portion  of  the  subsystem  contribution  that  applies 
to  the  system.  For  example.  Figure  3  shows  a  system  functional  makeup. 


LRU  3 


LRU  4 


LRU  6 


FIGURE  3.  PARTIAL  SYSTEM  FUNCTIONAL  MAKEUP 


Failures  will  propagate  through  the  system,  and  individual  detections 

of  a  single  failure  may  be  observed  at  the  outputs  of  individual  elements. 

For  example,  a  failure  in  LRU,  may  be  detected  at  the  output  of  LRU,  and 

l  o 

LRU,. .  Y  is  a  measure  of  the  propagation  effect  and  the  system  functional 


makeup.  If  the  i"‘  LRU  is  Isolated  in  testing,  then  Yj  ■  1.  For  the 

general  case,  it  is  a  function  of  feeds  and  topology.  A  first-order 
estimator  Is  given  by  the  complement  of  the  external  dependency  factor 
(EDF) : 


For  the 


_ ..  .  ,  number  of  inputs  to  subsystem  i 


1  1  total  subsystem  i  failure  list 

The  total  subsystem  i  failure  list  would  be  provided  by  the  FMECA  and 
would  include  Inputs,  so  that  an  LRU  with  25  possible  internal  failures 
and  5  Inputs  which  could  also  fall  would  be  estimated  at: 


m- 


=  (1  -  EDF. )  =  1  - 


25  +  5 


0.833 


.  J 


A  more  accurate  way  to  compute  Yj  would  be  to  analyze  the  system  FMECA  to 

tag  those  faults  which  actually  propagate.  The  methodology  described  in 
this  subsection  permits  the  FFD  parameters  to  be  built  up  from  sub¬ 
systems  to  systems. 

Note:  A  secondary  detection  such  as  a  backup  Indication  will  be 
counted  as  a  detection  by  the  supporting  subsystem.  It  will  be  termed 
"improper"  even  though  It  may  be  correct.  Thus,  proper/iraproper  is  only  a 
partitioning  and  does  not  Imply  correctness  of  the  detection.  A  prime 
example,  will  be  In  the  use  of  a  "centralized"  BITE  subsystem  that  may 
have  a  subsystem  contribution  to  FFD  of  1.0,  consisting  mostly  of  Improper 
detections. 


5.3  FRACTION  OF  FAULTS  ISOLATED  (FFI) 

Fraction  of  faults  Isolated  represents  the  meat  of  the  maintenance 
activity.  Its  Ideal  value  is  also  unity.  We  will  discuss  a  system- level 
FFI,  then  subsystem  values  of  FFI,  and.  finally,  a  system/ subsystem 
relationship. 

5.3.1  FFI  -  System 

The  set  theory  system  definition  derived  for  fault  Isolation  as  taken 
from  Appendix  B  Is: 

Fault  Isolation  -  NSM  identifies  all  failed  units 
within  the  system.  An  attempted  isolation  can  have 
any  of  the  following  results: 

-  Proper  Fault  Isolation  -  Only  and  all  failed  units 
are  Isolated. 

-  Improper  Fault  Isolation  -  All  but  not  only  failed 
units  are  Isolated. 

-  No  Fault  Isolation  -  Other  combinations  that  occur. 

Including  only  but  not  all  failed  units. 

To  relate  this  to  FFI.  we  will  again  normalize  by  the  faults  within 
the  system.  The  fraction  of  faults  isolated  at  the  system  level  will  then 
be  given  by:  “the  ratio  of  NSM  isolations  to  faults  within  the  system." 
(FFI  may  also  be  proper  or  improper.) 

From  the  Appendix  B  modified  state  representation,  this  definition  trans¬ 
lates  algorithmically  to: 

FFI  “  (ema  -  pIcnd) 


(Equation  55  of  Appendix  B) 


The  only  new  term  here  Is  the  IFI  ,  or  system-generated  fault 

Isolations  that  can  be  measured.  0  is  again  empirical,  as  discussed  In 

Section  5.2.1.  A  problem  unique  to  FFI  Is  the  breakdown  between  proper 

and  improper.  Since  FI  includes  both,  a  system  with  high  FFI  may  gen- 

s 

erate  a  large  number  of  RTOK  events.  To  create  a  term  less  sensitive  to 
this  problem,  we  develop  an  FFI^: 


HTVetces. 

IMA  -  p£CND 


'IFIS  -  IRTOK^ 
k  IMA  -  0ICND  > 


(Equation  56  of  Appendix  B) 


This  partially  compensates  for  improper  fault  Isolation,  as  discussed  In 
detail  In  Appendix  B. 

5.3.2  FFI  -  Subsystem 

The  set  theory-derived  definition  for  fault  Isolation  is  identical  to 
the  system-level  definition.  The  subsystem  algorithms  will  then  be: 


ss  (IMA  -  0ICND), 


SFIss  -  l!mKss 
(JMA  -  SZCND) 


Both  of  these  values  can  be  converted  to  subsystem-contribution  values. 
Let  Kj  be  the  ratio  of  faults  In  the  1th  subsystem  to  total  faults.  Then 


*1  J 


(IMA  -  0lCND)s# 
IMA  -  0ICND 


5.3.3  FFI 


'stem  Relatlonshi 


The  definitions  of  system  and  subsystem  offer  a  compatibility,  so 
that  the  final  relationship  is  given  by: 

FFI  -  l  (K  x  FFI  )  -  l  (FFI  ) 

SS  1  SSj  ss  ssc  1 

and 

FFI  *  X  (K  X  FFI  )  »  l  (FFI  ) 
p  SS  1  P’SS^  SS  p.SSC  1 

where  can  be  directly  computed.  The  methodology  described  In  this 
subsection  allows  the  buildup  of  a  system  FFI  from  subsystem  values. 


5.4  FRACTION  OF  FALSE  ALARMS  (FFA) 

FFA  represents  the  wasted  action  of  maintenance  activity.  Its  Ideal 
value  Is  zero.  We  will  first  discuss  a  system-level  FFA.  then  a  subsystem 
FFA,  and.  finally,  a  relationship  between  the  system  and  subsystem. 

5.4.1  FFA  -  System 

The  set  theory  system  definition  for  false  alarm  as  taken  from 
Appendix  B  Is: 

False  Alarm  -  There  Is  Indication  of  a  failure  In  the 
system  where  none  exists. 

To  relate  this  to  FFA.  we  must  normalize  by  some  factor.  If  we  use 
system  faults  as  we  did  with  the  two  previous  measures,  we  will  have  an 
111-deflned  parameter  In  that  false  alarms  may  exceed  actual  faults, 
giving  a  value  of  FFA  greater  than  1.  For  these  reasons  the  normalizer 
for  FFA  Is  the  sum  of  false  alarms  and  faults.  The  discussion  of 
relevancy  Is  particularly  important  to  false  alarms  as  pointed  out  in 
Chapter  One  and  Appendix  B.  Of  note  Is  the  exclusion  of  the  so-called 
“nuisance"  false  alarm,  which  Is  an  indication  that  Is  noted  and  then 
Ignored  (not  causing  a  maintenance  action). 

The  fraction  of  false  alarms  at  the  system  level  will  then  be  given 
by: 


The  ratio  of  the  system- level  false  alarms  to  the  sum 
of  the  faults  In  the  system  and  false  alarms. 


From  the  Appendix  B  state  representation,  this  can  be  algorithmically 
represented  by: 
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(Equation  57  of  Appendix  B) 


Again.  B  is  the  only  unmeasurable  quantity,  and  it  should  be  familiar  by 
non  because  It  appears  In  each  of  the  testability  parameters  to  be 
computed.  Of  particular  note  is  that  false  alarms  may  be  specialized  to 
system-generated  or  operator-generated  as  desired: 


FPAj 


(B£CND) j 
£MA 


where  (p£CND) ^  measures  the  jth  component  of  false  alarms  (e.g., 

system/opera tor /BIT) ,  and  FFA^  is  the  contribution  of  the  jth  component  to 
FFA. 


5.4.2  FFA  -  Subsystem 

The  subsystem  definition  for  false  alarm  as  taken  from  the  set  theory 
derivation  In  Appendix  B  Is  given  by: 

False  Alarm  -  There  Is  Indication  of  a  failure  In  the 
subsystem  where  there  Is  none  In  the  system. 

To  relate  this  to  FFA.  we  must  normalize  as  discussed  In  Section 
5.4.1.  so  that  the  subsystem  FFA  is  given  by: 

The  ratio  of  the  subsystem  false  alarms  to  the  sum  of 
the  faults  In  the  system  and  false  alarms. 


From  the  modified  state  representation,  this  definition  translates 
algorithmically  to: 


Subsystem  contribution: 


FFA 


ssc 


(BJCND) 
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This  equation,  of  course,  requires  us  to  partition  the  data  on  the 
basis  of  subsystem  properties: 

Subsystem  FFA: 

FFASS-  <IMA)SS 


These  measures  are  easiest  to  obtain  for  depot  and  intermediate  main¬ 
tenance,  where  the  individual  unit  Is  the  source  of  the  maintenance  action. 


5.4.3  FFA 


stem  Relatlonshl 


The  definitions  of  system  FFA  and  subsystem  FFA  can  be  related  by 
Equation  53  of  Appendix  B  as: 


I  /ffa  \  - 
1-1  \  ss/l 
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where  represents  the  cross-detection  of  subsystem  j  by  subsystem  1 

and  t -  0.  This  is  precisely  the  same  detection  problem  discussed  In 

Section  5.2.3.  and  It  reduces  false  alarms  at  the  system  level  by  a  factor 
applied  to  each  of  the  submeasures. 

This  equation  can  be  further  expanded  as  follows: 

aK)>  '  “  [JEJJ 

ss)i  111  j,K)k  1  Iffl  ' 

FFA  -  l  (FFA  J  -  I  5  (FFA  J 

1-1  \  SS/i  1-1  iV  SS/i 

where  ^  Is  a  modified  coefficient  of  the  subsystem  false  alarms  for  each 
of  the  1  subsystems  and  is  given  by 
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The  preceding  equation  can  be  written  as 


ihh  (Fra J4 


where  is  the  fraction  of  the  1^  subsystem  FFA  not  related  to  a 
cross -detect ion  problem  applied  at  the  system  level.  Figure  3  can  be 


used  to  Illustrate  this  factor.  A  detection  In  the  subsystem  LRUs  may  be 
caused  by  a  failure  In  LRU^,  but  at  the  LRU^  subsystem  level  It  will 

appear  as  a  false  alarm.  This  Is.  in  fact,  related  to  the  factor  devel¬ 
oped  In  Section  5.2.3  for  FFD,  so  that: 

4  a  Y  a  (1  -  EDF  ) 

1  1  1 


which  may  be  estimated  from  the  FMECA  as  discussed  In  Section  5.2.3. 

The  methodology  discussed  in  this  subsection  allows  the  buildup  of  a 
system  FFA  from  subsystem  values. 


5.5  ALGORITHMIC  APPLICATION  SUMMARY 

An  algebra  has  been  developed  to  permit  the  definition  of 
organizational- level  testability  attributes  (FFD,  FFI.  and  FFA).  This 
algebra  also  permits  a  buildup  from  subsystems  to  systems  so  that 
lower-level  analyses  can  be  combined  to  give  an  estimate  of  system 
performance.  Three  parameters  are  key  to  the  analysis  process.  At  both 
the  system  and  subsystem  levels  we  must  empirically  determine  the 
coefficient  fi  that  relates  the  "cannot  duplicate”  events  to  false  alarms. 
At  the  subsystem  level  we  need  to  determine  the  coefficient  Y  that 
allocates  the  portion  of  the  subsystem  contribution  that  applies  to  the 
system,  and  the  coefficient  4  that  represents  the  false-alarm/detection 
cross-talk  between  subsystems.  These  latter  two  may  In  fact  be  the  same 
parameter.  Estimators  are  provided  for  Y  and  4,  but  all  three  should  be 
empirically  derived  from  field  data. 

If  these  algorithms  can  be  used  to  measure  the  parameters  of  interest 
then  prediction  can  be  developed  through  a  number  of  cause-and-effect 
processes.  Including  regression  and  analytic  and  theoretical  development. 


CHAPTER  SIX 


FEASIBILITY  OF  PREDICTOR  DEVELOPMENT 


This  chapter  discusses  the  feasibility  of  developing  predictors  for 
the  three  measures.  The  basic  decision  process  is  defined,  and  then  each 
major  decision  made  Is  discussed  In  detail. 


6.1  THE  FEASIBILITY  DECISION  PROCESS 

To  make  the  decision  concerning  feasibility  as  objective  as  possible, 
we  have  developed  a  decision  process  that  asks  key  questions  and,  on  the 
basis  of  the  response  to  the  questions.  Indicates  whether  FFA,  FFD,  and 
FFI  can  be  practically  predicted,  require  further  study,  or  are  impracti¬ 
cal.  The  generalized  decision  process  Is  shown  In  Figure  4. 

Starting  with  each  mathematical  model  (discussed  In  Appendix  B),  we 
ask,  "Can  we  measure  the  parameters?**  This  Is  a  function  of  the  elements 
of  the  parameter  as  defined  In  the  mathematical  model  and  of  the  mainte¬ 
nance  data  collection  and  reporting  system.  If  the  elements  cannot  be 
measured  directly,  we  may  be  able  to  Infer  the  elements  (or  the  measure 
directly)  from  the  data.  For  example,  we  may  be  able  to  determine  (from 
outside  sources)  an  empirical  coefficient  that  permits  Inferring  one  of 
the  elements  of  a  measure.  If  not,  the  decision  process  asks  if  we  can 
develop  a  data  collection  program  that  can  provide  the  data.  If  we  can¬ 
not.  there  is  no  need  to  continue;  it  Is  Impractical  to  measure  that  par¬ 
ticular  parameter.  If  a  data  collection  system  can  be  developed,  we  may 
continue  to  the  next  level  only  If  the  system  Is  simple  enough  to  fit 
within  the  Phase  II  effort.  If  It  Is  not.  we  will  declare  that  measure 
“not  practical  now." 

If  we  successfully  traverse  the  "measurability*'  portion  of  the  deci¬ 
sion  process,  we  next  assess  the  existence  of  design  data  that  can  differ¬ 
entiate  various  systems;  and  we  ask  the  same  types  of  questions  asked  in 
the  measurability  loop:  If  data  do  not  exist,  can  they  be  inferred?  If 
not.  can  we  set  up  a  design  data  program  that  could  produce  these  data? 

Note  that  we  must  reach  the  conclusion  of  "impractical"  if  we  have 
answered  no  to  all  these  questions;  to  have  done  so  Implies  that  there  are 
no  distinctions  between  electronic  equipment  --  i.e.,  no  peculiarities 
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about  the  design  of  box  A  that  permit  It  to  be  Identified  as  separate  and 
distinct  from  box  B.  Intuitively,  this  Is  not  the  case;  there  should 
always  be  some  design  factor  that  makes  boxes  different.  If  only  in  func¬ 
tion.  However,  it  should  be  recognized  that  the  key  issue  Is  quantifi- 
ablllty.  i.e.,  some  way  to  measure  design  difference.  This  is  necessary 
since  we  cannot  develop  useful  predictors  when  the  variables  are  totally 
subjective  or  unquantified. 

If  the  decision  process  has  shown  that  the  parameter  is  measurable, 
and  that  there  are  quantifiable  design  differences  between  equipments,  the 
next  step  Is  to  determine  if  relationships  exist  between  the  parameter  and 
the  design  data.  If  none  exists,  we  must  again  conclude  that  it  Is  lmprac 
tlcal  to  build  a  predictor  of  that  parameter. 

The  last  level  of  decision  checks  to  determine  If  there  are  un¬ 
explained  variances  in  the  predictor  of  FFI,  FFD,  or  FFA;  if  there  are 
not,  we  declare  it  practical  to  develop  a  predictor  for  that  testability 
measure.  If  unexplained  variances  exist,  we  may  be  able  to  quantify  them 
with  some  other  measure;  If  we  can,  we  will  declare  the  development  of 
that  predictor  practical;  If  we  cannot,  we  say  that  the  measure  requires 
further  study,  either  to  quantify  the  unexplained  variance  or  to  determine 
If  the  variances  are  critical. 

This  decision  process  was  conducted  for  each  of  the  three  measures  of 
Interest  —  FFI.  FFD.  FFA  (or  FAR).  Note  that  we  have  used  the  term 
“practical"  as  opposed  to  “feaslble“  up  to  this  point.  The  reason  is  that 
we  should  differentiate  between  the  practicality  of  a  predictor  of  one 
measure  of  testability  and  the  “feasibility"  of  continuing  the  entire  pro¬ 
cess  In  light  of  the  Interrelationship  of  the  parameters  as  shown  In  Chap¬ 
ter  Five  and  Appendix  B. 

We  have  developed  a  decision  approach,  which  includes  consideration 
of  the  Importance  of  the  measures  and  the  practicality  of  predicting  each 
measure. 

ve  believe  that  FFA  (or  FAR)  Is  central  to  the  feasibility  of  the 
overall  predictor  model,  especially  In  light  of  its  appearance  in  both  of 
the  other  two  measures.  FFD  is  a  critical  parameter  because  of  Its  Impact 
on  readiness  and  mission  success  —  Ideally,  we  should  detect  failures  In 
the  preflight  period  so  that  missions  can  proceed  at  full  capability.  The 
Isolation  problem  is.  in  that  regard,  slightly  less  critical.  Red  Ball  or 
Red  Streak  teams  can  always  perform  a  wholesale  system  swapout  and  let 
Intermediate  maintenance  perform  the  actual  fault  Isolation. 

The  following  sections  track  the  decision  process  for  the  three  test¬ 
ability  measures. 


6.2  MEASURABILITY 

The  first  level  of  the  decision  process  is  measurability.  The  three 
measures  and  the  mathematical  expressions  are  summarized  in  Table  4;  the 
details  of  their  derivation  are  examined  In  Appendix  B. 
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We  have  shown  FFI  as  the  “total"  FFI  measure,  disregarding  any  dis¬ 
tinction  between  proper  and  Improper  fault  Isolation.  This  distinction 
will  be  discussed  in  Section  6.2.3  in  more  detail,  because  it  poses  inter¬ 
esting  measurability  problems. 

Several  observations  should  be  made:  Employing  the  set  theory  equa¬ 
tions  requires  a  data  collection  system  that  can  classify  each  maintenance 
action  and  failure  Indication  exactly:  that  is,  it  assumes  perfect  infor¬ 
mation.  For  that  reason,  the  set  theory  equations  are  not  useful  from  a 
"measurability"  viewpoint:  rather,  the  set  theory  model  was  useful  in 
defining  the  assumptions  that  were  Implicitly  made  In  the  other  two  models 
but  were  not  readily  Identifiable.  Hence,  the  set  theory  equations  act  as 
a  "consistency  check"  of  the  other  two  formulations,  and  they  will  not  be 
further  addressed  in  this  chapter. 

Note  also  that  p  CND  appears  In  every  testability  parameter  in  the 
state  and  flow  models  and  is  thus  the  major  key  to  the  success  of  any  pre¬ 
dictor  model.  If  we  cannot  measure  and  "predict"  the  "cannot  duplicate" 
term,  the  other  two  measures  will  be  in  error.  For  this  reason,  we  will 
discuss  FFA  (FAR)  first. 

6.2.1  Fraction  of  False  Alarms  (FFA) 


As  can  be  seen  in  Table  4,  FFA  is  structurally  the  simplest  of  the 
measures  and  conceptually  the  easiest  to  measure. 


The  maintenance  reporting  system  has  a  column  in  which  the  technician 
enters  a  "how  malfunction  code,"  and  a  list  of  acceptable  codes  is  given. 
One  of  these  is  799,  or  "cannot  duplicate,  bench  check  good,"  or  "no  fault 


found.”  Consequently,  by  counting  the  number  of  occurrences  of  799  for  a 
given  LRU  work  unit  code  over  a  given  period,  and  counting  the  total  main¬ 
tenance  actions  (job  control  numbers)  generated  for  that  LRU  over  the  same 
period  of  time,  we  obtain  a  measure  of  the  fraction  of  maintenance  actions 
that  result  In  CND  (PCND). 

Converting  PCND  to  our  fraction  of  false  alarms  requires  that  we  be 
able  to  determine  the  parameter  3,  which  Is  the  proportion  of  CNDs  that 
are  actually  false  alarms.  This  may  be  possible  where  "bad  actors"  are 
carefully  tracked.  In  that  case,  a  CND  that  was  a  repeat  CND  on  that  par¬ 
ticular  serial  number  LRU  would  be  Interpreted  a  failure  whose  Isola¬ 
tion  escaped  normal  troubleshooting.  Extraordinary  measures,  not  part  of 
normal  system  maintenance,  would  be  required  to  verify  and  isolate  the 
failure.  In  any  case,  determining  3  requires  careful  tracking  of  each 
CND  to  weed  out  the  "bad  actors." 

Several  of  the  operational  units  that  we  visited  reported  that  they 
employ  some  sort  of  "bad  actors"  program.  Such  programs  tend  to  be  ad  hoc 
and  are  almost  universally  manual,  existing  In  the  form  of  entries  in  the 
repair  shop's  equipment  logs  or  similar  files.  Consequently,  these  data 
are  not  amenable  to  computerized  analysis  and  should  be  viewed  as  a  last- 
resort  source  of  data. 

The  C-5  Malfunction  Analysis  Detection  and  Recording  System  (MADARS), 
on  the  other  hand.  Is  highly  automated  and  appears  to  be  robust  enough  to 
Infer  a  3  factor  for  at  least  some  of  the  C-5  avionics.  There  Is  a  “bad 
actors"  analysis  program,  and  the  requisite  serial  number  tracking  capa¬ 
bility.  In  addition,  a  quick  look  Indicates  that  It  would  be  feasible  to 
establish  a  link  Into  the  MADARS  data  base  (without  disturbing  day-to-day 
operations)  for  any  special-purpose  processing  that  might  be  needed. 

A  second  factor  tends  to  complicate  the  determination  of  FFA.  In 
many  cases,  maintenance  organizations  use  a  summary  analysis  of  the  AFTO 
349  data  to  measure  performance  of  the  maintenance  activities.  A  possible 
result  Is  that  activities  with  numerous  CNDs  will  be  judged  somehow  less 
capable  than  those  with  few  CNDs.  Maintenance  technicians  will  thus  be 
reluctant  to  use  the  799  "how  mal"  code  —  rather,  they  will  find  some 
adjustment  to  make  and  will  report  that  action. 

The  problem  Is  not  a  simple  one;  It  Is  natural  that  some  "measure”  of 
technical  performance  be  developed  for  maintenance  activities,  and  CND 
rate  would  seem  to  Indicate  the  diligence  of  technicians  In  fault 
verification/isolation.  On  the  other  hand,  there  are  "genuine"  "cannot 
duplicates”  that  can  be  Indicative  of  systemic  problems  (e.g..  BIT  inade¬ 
quacies.  test  equipment  deficiencies,  "true"  lntermlttents) ,  and  It  should 
be  a  goal  of  the  maintenance  system  to  highlight  them  so  that  they  can  be 
addressed,  thereby  ultimately  Improving  overall  readiness. 

There  also  appear  to  be  some  CND  biases  In  the  opposite  direction. 

One  base  seemed  to  be  using  CND  as  the  "how  mal"  whenever  there  was  no 
"remove  and  replace";  reseating  connectors  when  a  fault  was  indicated 
resulted  In  a  good  bench  check,  so  "how  mal”  was  recorded  as  799. 


Finally,  Gemas1  reported  that  CND  rates  have  varied  as  much  as  50 
percent  between  bases  for  the  same  LRU.  The  reasons  for  this  variation 
may  be  exemplified  by  the  preceding  observations.  In  any  case,  obtaining 
an  adequate  measure  of  FFA  will  be  dependent  on  a  good,  rich  data  base 
with  sufficient  analysis  and  tracking  capabilities  to  permit  estimating 
the  fractions  of  CND  that  are  false  alarms .  There  appears  to  be  hooe 
for  this  in  the  NADARS. 

The  preceding  comments  apply  to  both  the  state  model  and  flow  model 
representations  of  FFA.  In  one  case  the  measured  data  are  a  closest  count 
of  events;  In  the  other  the  measured  parameter  Is  a  rate  —  maintenance 
actions  per  month,  or  whatever  time  period  appears  useful.  Since  mainte¬ 
nance  actions  are  tracked  by  job  control  numbers,  which  In  turn  contain 
the  operation  date,  rate  data  are  available  (and  subject  to  the  biases 
already  discussed).  In  addition,  many  bases  use  long-term  rates  for  man¬ 
agement  reporting,  such  as  CND  rate  and  repair  rate.  However,  to  convert 
these  data  to  a  measure  of  FFA,  either  a  functional  form  of  the  data  is 
needed  so  that  the  Integrations  specified  in  Table  4  can  be  performed  or 
the  data  must  be  stationary  over  some  period.  The  former  requirement  Is 
very  restrictive,  and  probably  unachievable.  The  latter  Is  tantamount  to 
saying  that  the  rate  of  maintenance  actions  and  rate  of  CNDs  are  constant 
over  time  (for  a  given  LRU).  Although  this  Is  unlikely  to  be  true.  It  can 
be  assumed  that  when  taken  over  a  long  enough  period,  the  rate  may  be 
represented  by  some  average  rate  over  a  period  T.  The  equations  In  Table 
4  then  provide  the  same  result  as  the  state  model: 

3Jt  CND  dt  3  x  CND  x  t 

JT  MA  dt  MA  x  t 

3  x  Total  CNDs 
Total  MAs 

P  lT  CND 
*  lT  MA 

Although  the  flow  model  formulation  for  FFA  does  not  provide  any 
clearly  superior  or  advantageous  way  to  measure  FFA,  It  does  emphasize  the 
need  to  set  the  time  span  long  enough  to  capture  any  periodicity  in  the 
data. 

To  summarize,  fraction  of  false  alarms  appears  to  be  indirectly  mea¬ 
surable.  The  solution  of  FFA  that  can  be  directly  measured  is  the  number 
of  CNDs  divided  by  total  maintenance  actions,  or  "fraction  of  CND”  (FCND) . 


x "Aircraft  Avionics  System  Maintenance  Cannot  Duplicate  and  Retest-OK 
Analytical  Source  Analysis.”  Capt.  G.  L.  Gemas,  AFIT  Master's  Thesis, 
September  1983. 


If  the  parameter  3  can  be  determined  by  careful  analysis  of  “bad  actors." 
then  FFA  *  3  x  FCND.  Further  Investigation  Into  the  factors  driving  3 
requires  access  to  an  extensive  data  source  and  the  capability  to  tie 
maintenance  actions  to  specific  LRUs  (by  serial  number).  Even  If  3 
cannot  be  determined,  the  parameter  FCND  is  a  useful  one,  since  It  does 
relate  to  testability  design,  and  “cannot  duplicate"  events  have  Impacts 
on  operational  readiness  similar  to  those  caused  by  false  alarms.  Note 
that  at  the  organizational  level,  every  false  alarm  should  be  classified 
as  a  CND;  however,  not  every  CND  Is  a  false  alarm.  If  most  CNDs  are  in 
fact  due  to  false  alarms  (l.e..  3  Is  close  to  1.0),  then  one  might 
consider  an  estimate  of  FCND  as  an  upper-limit  estimator  for  FFA. 

6.2.2  Fraction  of  Faults  Detected  (FFD) 

The  state  representation  for  fraction  of  faults  detected  Is 

IMAS  -  3s2CNDs 
IMA  -  0ICND 

A  flow-model  representation  has  the  analog  form 

f(ms  -  3SCI^DS) 

•  • 

J (MA  -  3  CND) 

As  discussed  earlier,  the  flow  representation  can  be  reduced  to  the 
state  representation  by  taking  the  integration  over  a  large  enough  window 
that  the  rates  can  be  approximated  by  constants,  and  this  implies  that  we 
must  use  a  similarly  “large  enough"  window  in  summing  events  in  the  state 
representation.  In  both  representations,  the  key  factor  for  measurability 
is  the  ability  to  separate  system-generated  events  from  non-system 
(operator) -generated  events.  This  division  between  "system-generated” 
("normal  system  maintenance")  and  "non-systera-generated"  (“other  than  nor¬ 
mal  system  maintenance" )  is  a  crucial  one.  As  discussed  earlier,  the  ulti¬ 
mate  goal  of  this  study  is  to  provide  a  predictor  that  equipment  developers 
can  use  to  evaluate  how  well  the  NSM  (the  set  of  built-in-test  software, 
check  lists,  and  Technical  Orders)  that  they  provide  with  the  equipment 
can  detect  and  isolate  failed  conditions.  To  develop  such  a  predictor,  we 
must  be  able  to  determine  whether  or  not  events  like  failure  detections 
and  fault  isolations  are  due  to  NSM.  We  must  also  be  able  to  separate 

CND  into  CNDs  and  "bad  actors."  This  ability  is  provided  by  factor  3_» 
s  s 

which  is  analogous  to  3.  discussed  in  Section  6.2.1. 


In  the  normal  AFTO  349  maintenance  documentation,  there  is  no  data 
field  that  can  reliably  indicate  whether  a  fault  was  detected  by  the  oper¬ 
ator  or  by  the  BIT.  The  "when  discovered"  code  indicates  "in-flight"  or 
“ground,"  but  it  is  difficult  to  argue  that  "in-flight"  implies  "pilot- 
detected"  or  even  that  "pilot-detected"  means  "other  than  normal  system- 
detected,"  since  the  pilot  may  have  been  using  a  check  list  with 


maintenance-related  actions.  The  only  way  to  resolve  this  situation  Is  to 
correlate  pilot  debriefs  with  the  349s.  In  general,  this  is  difficult  to 
do  In  a  large-scale  automated  fashion. 

However,  the  C-5  MADARS  offers  one  approach,  and  a  special  BIT  study 
currently  under  way  in  the  F-16  offers  another. 

The  C-5  MADARS  captures  malfunction  alarms  in  a  series  of  aircraft 
avionics;  after  the  mission  Is  completed,  the  MADARS  data  are  correlated 
with  pilot  debriefs.  Clearly,  any  maintenance  action  resulting  from 
MADARS-noted  real functions/ faults  are  system-generated. 

The  F-16  BIT  reports  malfunctions  to  the  pilot  with  a  Maintenance 
Fault  Listing  Summary  (MFLS)  code.  During  debrief,  these  MFLS  indications 
are  passed  to  the  maintenance  crew.  If  a  maintenance  action  is  Initiated 
for  which  there  is  an  MFLS  and  the  "when  discovered"  code  is  “in-flight.” 
we  can  assume  that  action  was  system-generated. 

Both  of  these  methods  for  determining  MA  and  CND  have  some  defi- 
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clencles.  The  C-5  MADARS  examines  only  a  limited  set  of  C-5  avionics  — 
principally  the  INS  and  the  Central  Air  Data  Computer  —  that  would  be 
routinely  found  in  many  aircraft.  Consequently,  the  range  of  data  col¬ 
lected  would  be  limited.  However,  the  entire  AFTO  349  reporting  system  at 
C-5  maintenance  complexes  is  automated,  making  general  tracking  of  main¬ 
tenance  actions  potentially  simpler. 

The  F-16  data  set  is  richer  than  the  C-5  in  that  it  samples  a  larger 
set  of  avionics  types;  however,  the  separation  of  "system-"  and  "operator-" 
discovered  faults  depends  on  the  reliability  of  the  pilot  debrief  process; 
if  the  MFLS  codes  are  not  recorded  against  the  resulting  maintenance 
action,  a  system-generated  action  or  detection  will  be  erroneously  counted. 
This  problem  can  be  overcome  through  appropriate  training  and  follow-up 
monitoring  of  the  operators  and  the  maintenance  personnel. 

To  summarize,  fraction  of  faults  detected  requires  measuring  the  CNDs 
that  are  false  alarms.  These  are  subject  to  the  biases  and  measurement 
problems  addressed  in  Section  6.2.1.  In  addition,  FFD  requires  a  way  to 
separate  system-discovered  CND  and  MAs  from  operator-discovered  events. 
There  is  no  formal  way  to  do  this  in  the  current  standard  maintenance 
reporting  system,  but  both  the  C-5  MADARS  and  the  F-16  appear  to  have 
potential  for  a  specially  constructed  data  collection  effort. 

6.2.3  Fraction  of  Faults  Isolated  (FFI) 


These  represent  " total “  fault  Isolations,  without  regard  to  whether  the 
Isolation  was  correct  and  minimally  sufficient  (the  fault  was  isolated 
only  to  units  that  had  failed,  and  not  to  an  ambiguous  group  of  units). 
To  account  for  imperfect  fault  isolation,  we  also  developed  an 

FFI  ; 
performance* 


FFI 


P 


!FIs  -  £RTOKs 
IMA  -  0JCND 


Note  that  we  have  reduced  the  fault  Isolations  by  the  RTOKs  generated 
at  the  next  level  of  maintenance.  If  all  fault  Isolations  are  correct. 
RTOK  will  be  zero. 

As  with  FFD,  we  must  measure  the  CNDs  that  are  false  alarms,  with  all 
the  attendant  difficulties,  as  discussed  in  Section  6.2.1.  In  addition, 
we  must  measure  fault  isolations  by  the  maintenance  system. 

This  parameter  can  be  obtained  from  the  AFTO  349s  under  the  “Action 
Taken"  column.  Certain  of  the  AT  codes  imply  a  fault  isolation,  for  exam¬ 
ple.  "F“  (repaired)  or  "R"  (removed  and  replaced).  Counting  system  fault 
isolations  then  reduces  to  counting  maintenance  actions  for  which  the  AT 
code  is  in  the  set  of  "fault-isolated"  codes.  However,  without  some 
indication  of  the  method  of  fault  isolation  (normal  system  means  or  not), 

we  cannot  measure  FFI  . 

s 

Measuring  penalized  FFI  is  even  more  difficult,  because  we  must 
determine  the  RTOKs  from  the  next  level  of  maintenance  for  a  given  system. 
Here  the  data  collection  window  becomes  even  more  critical,  since  the 
RTOKs  will  be  “delayed"  by  twice  the  pipeline  time  from  one  maintenance 
level  to  the  next.  If  the  data  are  stationary,  this  time  shift  will  not 
matter,  since  the  rates  of  RTOK  will  be  constant.  He  will  discuss  sta- 
tionarity  of  data  in  Section  6.4.2. 

6.2.4  Measurability  Summary 

On  the  basis  of  the  preceding  discussions,  we  draw  the  following 
conclusions: 

-  The  state  model  equations  are  more  directly  useful  for  measuring 
the  parameters  of  interest.  The  maintenance  data  collection  sys¬ 
tems  in  place  are  better  suited  for  event  counting  than  for  rate 
measurement . 

-  The  most  critical  parameter  is  FFA,  since  its  constituent  terms 
appear  in  the  other  two  measures  (FFD  and  FFI). 

-  FFA  cannot  be  directly  measured  with  current  data  collection  sys¬ 
tems.  since  there  is  no  mechanism  for  determining  p.  the  fraction 
of  CNDs  that  are  actually  false  alarms. 


FFD  cannot  be  directly  measured  with  current  data  collection  sys¬ 
tems,  since  there  is  no  reliable  way  to  identify  system-discovered 
failures. 


-  A  data  collection  and  analysis  program  could  be  structured  to  pro¬ 
vide  the  necessary  data  for  FFD  and  FFA  measurements,  at  least  for 
a  limited  set  of  avionics.  FFI  would  also  be  measurable. 

In  summary,  the  first  level  of-  the  decision  process  suggests  that  the 
three  testability  parameters  are  not  practical  now.  A  special  data  col¬ 
lection  and  analysis  effort  must  be  established  before  a  predictor  model 
can  be  developed. 

We  will  continue  through  the  decision  tree  In  the  remaining  sections 
of  this  report  as  If  we  could  measure  the  parameters  to  determine  what  (If 
any)  other  stumbling  blocks  might  exist. 


6.3  DESIGN  PARAMETERIZATION 

This  section  addresses  the  second  level  of  the  feasibility  decision 
process,  the  existence  of  design- related  parameters  that  differentiate 
between  equipments.  In  essence,  we  must  answer  the  question  “What  attri¬ 
butes  does  a  piece  of  hardware  have  that  could  Influence  its  testability 
characteristics  from  all  other  hardware?"  Whether  or  not  these  attributes 
have  a  quantifiable  effect  on  FFA.  FFD.  or  FFI  Is  an  Issue  that  will  be 
resolved  In  the  third  level;  however,  we  will  attempt  In  this  section  to 
Identify  parameters  that  can  reasonably  be  expected  to  display  a 
relationship. 

Intuitively,  these  equipment  design  parameters  must  provide  some  in¬ 
dication  of  the  Interrelationship  of  the  elements  of  the  system  in  ques¬ 
tion  (e.g.,  3RD  Interrelationship  for  an  LRU.  component  interrelationship 
for  SRUs ) .  Given  the  success  of  testability  evaluation  models  such  as 
LOGMOD,  STAMP,  and  others.  It  is  clear  that  such  parameters  must  exist. 

System  testing  Is  an  attempt  to  determine  the  state  of  the  system. 
System  testability  is  a  measure  of  the  ability  to  correctly  determine  the 
system  state.  Let  us  examine  a  system  with  only  two  states  (failed  and 
good)  and  one  test.  Table  5  shows  the  possible  outcome  of  such  testing. 


TABLE  5.  SIMPLIFIED  SYSTEM  TEST 
MATRIX 


System  State 


Test 

Outcome 

Failed 

Good 

Failed 

Detection 

False  Alarm 

Good 

Nondection 

No  Fault 

Isolation  Is  an  extension  of  this  detection  problem  to  Its  lowest 
level.  Factors  that  may  Influence  the  detectability  of  failures  or  the 
Improper  indication  of  failures  may  include  the  following: 

-  System  complexity  (number  of  possible  states)  -  In  general,  the 
greater  the  number  of  elements  In  a  system,  the  greater  the  number 
of  possible  states.  This  is  further  complicated  by  the  types  of 
elements  present.  For  example,  a  system  composed  of  N  two-terminal 
devices  is  inherently  far  simpler  than  one  composed  of  N  VHSIC 
devices.  Some  of  the  states  that  may  be  unanticipated  may  be 
determined  In  testing  to  be  false  alarms.  From  a  complexity  stand¬ 
point  one  would  expect  digital  systems  to  have  a  lower  detection 
capability  and  a  higher  false-alarm  rate. 

-  Structure  -  The  number  of  paths  that  lie  between  a  stimulus  and 
Its  response  will  be  related  to  the  ambiguity  in  determining  the 
meaning  of  the  output.  Parallel  structures  should  then  lead  to 
Increased  false  alarms,  while  serial  structures  should  lead  to 
decreased  false  alarms.  Two  possible  measures  of  system  structure 
are  parallelism  (the  number  of  parallel  paths)  and  feedback. 

-  Number  and  sophistication  of  tests  -  Systems  that  are  pushing  the 
state  of  the  art  would  tend  to  have  higher  failure  rates,  and 
tests  would  be  developed  to  uncover  system  failures.  We  would 
expect  a  larger  number  of  tests  with  closer  tolerances.  Systems 
that  are  not  pushing  the  state  of  the  art  would  tend  to  have  lower 
failure  rates,  and  tests  would  be  developed  to  verify  proper  sys¬ 
tem  operation.  We  would  expect  a  lower  number  of  more  tolerant 
tests.  The  former  types  of  systems  would  then  tend  to  develop  a 
greater  false-alarm  potential.  The  latter  would  have  a  reduced 
false-alarm  potential  but  also  a  reduced  detection  capability. 

Many  other  factors  may  be  included,  such  as  the  following: 

-  Component  technology  (e.g..  digital,  analog,  special) 

-  Design  architecture  (e.g.,  function  interdependence,  interface 
complexity) 

-  Maintenance  architecture  (e.g.,  test  or  calibration  requirements) 

-  System  maturity 

These  will  be  examined  for  predictor  development  in  Phase  II. 

The  following  parameters  were  developed  on  the  basis  of  the  preceding 
discussion  as  a  vehicle  for  determining  the  feasibility  of  developing 
predictors. 


6.3.1  Mumber  of  Elements 


When  appropriately  normalized,  the  number  of  elements  Indicates  some¬ 
thing  about  design  complexity.  The  suggested  normalization  factor  is 
“ functional  elements."  Hence,  we  hypothesize  a  design  parameter  for  LRU 
testability  prediction  as 

_  total  number  of  SRUs  In  1th  LRU 
total  subfunctions  performed  by  LRU 


where  the  L  subscript  Indicates  an  LRU  measure. 

For  SRUs,  this  parameter  would  become 

number  of  components  In  SRU 

NCy,  * 

^  total  subfunctions  performed  by  SRU 
where  the  C  subscript  Indicates  a  component  measure. 

Hence,  In  the  LRU  case.  If  we  had  a  system  that  was  a  communications 
receiver/transmitter,  the  following  seven  system  subfunctions  might  be 
performed: 

-  RF  pre-amplification  -  Modulator 

-  Down-conversion  -  Power  amplifier 

-  Detection  -  Power  supply 

-  I/O  interface 

If  this  system  had  six  LRUs,  then  the  normalized  design  parameter 
would  be  6  +  7.  or  0.857. 

For  fault  Isolation,  It  appears  that  an  Ideal  value  of  this  parameter 
would  be  1.  By  determining  what  subfunction  Is  at  fault,  the  faulty  LRU 
Is  Immediately  determined.  Numbers  less  than  1,  while  still  allowing  un¬ 
ambiguous  LRU  Identification,  Imply  that  serviceable  functions  are  replaced 
unnecessarily,  causing  higher-than-necessary  cost  for  replacement  parts. 
Numbers  much  greater  than  1  tend  to  Indicate  large  ambiguity  groups  —  for 
example.  If  there  are  three  LRUs  for  each  subfunction,  identifying  the 
failed  subfunction  shows  only  that  there  Is  a  failure  In  a  set  of  three 
LRUs.  The  maintenance  process  must  expend  more  time  and  resources  Identi¬ 
fying  which  of  the  three  LRUs  is  truly  faulty.  Alternatively,  all  three 
LRUs  might  be  replaced,  causing  a  high  "retest  OK"  rate  since  not  all 
three  of  the  "failed"  LRUs  truly  have  failed. 

Determining  a  value  for  this  design  parameter  may  not  be  a  simple 
matter.  It  requires  detailed  understanding  of  the  function  of  the  system. 
During  the  design  phase,  this  Is  not  a  difficult  challenge;  LRU  functions 
may  not  be  clearly  Identified  In  the  field  technical  manuals. 


6.3.2  Number  of  Test  Points 


The  number  of  test  points  provides  an  indication  of  the  degree  of 
access  to  the  subelements  for  test  purposes.  Intuitively,  the  more  sub¬ 
elements  (SRUs)  or  subfunctions  an  LRU  has,  the  more  test  points  it  should 
have.  Consequently,  this  parameter  could  be  normalized  by  the  number  of 
SRUs  (TP)  or  by  the  number  of  functions  (TP) : 


number  of  test  points 
number  of  subfunctions 


or 


TPS 


number  of  test  points 
number  of  SRUs 


There  is  a  relationship  between  TFp  and  TPs'* 


TPg  x  NE^»  since  NEL 


number  of  SRUs _ 

total  subfunctions  in  LRU 


Increasing  TPp  or  TPg  should  reflect  improved  testability.  The  more 

test  points  provided  per  SRU  or  function,  the  better  the  ability  to 
Isolate  a  failure.  Detailed  design  documentation  is  necessary  to  deter¬ 
mine  this  parameter  accurately. 

6.3.3  Feedback 

Feedback  in  a  system's  design  begins  to  touch  on  the  "architecture'' 
of  the  system.  A  parameter  that  indicates  the  degree  of  feedback  present 
in  a  system  indicates  something  about  the  basic  "interconnectedness"  of 
the  subelements.  There  are  a  number  of  ways  in  which  feedback  can  be 
evaluated;  two  of  these  are  the  number  of  feedback  loops  (NFL)  and  the 
average  number  of  subelements  contained  (or  spanned)  in  a  feedback  loop. 
STAMP  refers  to  this  as  CFD  for  component  feedback  dominance.  These 
parameters  are  illustrated  in  the  sample  system  of  Figure  5.  Here,  then, 
is  one  feedback  loop,  formed  by  CP7  and  CP9,  so  that  NFL  =*  1,  CFD  *  2. 

6.3.4  Parallelism 


Parallelism  also  touches  on  the  "interconnectedness"  issue.  Systems 
with  no  parallelism  should  be  easily  fajjlt-isolated  (using  half-interval 
search,  for  example);  but  techniques  for  highly  parallel  systems  are  less 
well  known,  because  there  seems  to  be  no  clearly  apparent  "optimal  strat¬ 
egy"  for  fault  isolation.  It  should  be  noted,  however,  that  parallelism 
in  some  systems  makes  it  easy  to  detect  the  presence  of  faults.  The 
parameter  degree  of  parallelism  (DP)  can  be  expressed  as 

no  ,  number  of  parallel  paths 
op  *  number  of  SRUs 


Til 


FIGURE  5.  SAMPLE  SYSTEM 


A  first-level  approximation  of  this  can  be  obtained  by  counting  the 
maximum  number  of  paths  of  a  functional  or  SRU  block  diagram  and  dividing 
that  number  by  the  number  of  SRUs.  In  the  sample  system  of  Figure  5, 

DP  =  5  *  9.  A  purely  serial  system  would  have 


number  of  SRUs  ' 

which  tends  to  0  as  the  number  of  SRUs  Increases.1 
6.3.5  External  Dependency 

The  parameters  just  discussed  concentrate  on  "internal"  descriptions 
of  a  system.  Such  measures  seem  to  deny  the  relationship  between  a  sys¬ 
tem's  performance  and  Its  outside  interfaces.  It  Is  reasonable  that,  for 
example,  a  system's  false-alarm  rate  would  be  somehow  affected  by  Its 
dependency  on  external  sources  of  Information.  A  parameter  that  measures 
such  dependency  Is  external  dependency  (ED),  the  ratio  of  the  number  of 

*As  a  working  defintlon  for  this  feasibility  work,  the  number  of  paral¬ 
lel  paths  is  the  largest  number  of  lines  on  the  diagram  that  will  be  "cut 
by  any  Imaginary  vertical  line.  For  the  sample  system  this  is  five,  with 
the  imaginary  vertical  line  between  CP4  and  CP6.  It  is  recognized  that 
that  with  a  proper  redrawing  of  the  figure  the  number  of  lines  could  be 
four.  This  will  provide  some  variance  in  the  computation  of  DP.  but  this 
variance  should  be  small  in  all  but  those  systems  containing  few  SRUs. 


Input  signals  to  the  number  of  elements.  An  LRU  that  gathered  large  quan¬ 
tities  of  data  from  external  sources  and  had  a  single  SRU  would  have  a  high 
degree  of  external  dependency.  Thus 

_  ^  number  of  LRU  inputs 
number  of  SRUs 

6.3.6  Design  Parameter  Summary 

Table  6  summarizes  the  design-related  parameters  discussed  above; 
during  the  actual  development  of  a  predictor,  other  parameters  may  be 
developed  as  well.  Section  6.4  addresses  the  existence  of  a  relationship 
between  these  parameters  and  the  measures  FFA,  FFI,  and  FFD. 


Name 


TABLE  6.  DESIGN  PARAMETERS1 


Symbol 


Method  of  Calculation 


Normalized  Elements  NE 

Normalized  Test  TP 

Points 

Feedback  Loops  FL 

Component  Feedback  CFD 

Dominance 

Degree  of  DP 

Parallelism 

External  Dependency  ED 


_ number  of  SRUs _ 

number  of  functions  in  LRU 

number  of  test  points  In  LRU 
number  of  SRUs  In  LRU 

feedback  loops  in  LRU  block  diagram 

number  of  SRUs  in  feedback  loops 
total  number  of  SRUs 


number  of  parallel  paths  in  LRU  block  diagram 
number  of  SRUs  in  LRU 

number  of  inputs 
number  of  SRUs 


xHierarchial  decomposition  for  LRU  -»  SRU  testability.  To  convert  to  com¬ 
ponent  testability,  replace  "SRU"  with  "component"  and  "LRU”  with  "SRU."  To 
convert  to  system  testability,  replace  "LRU"  with  "system"  and  "SRU"  with  "LRU. 


6.4  RELATIONSHIPS 

We  noted  in  Section  6.2  that  the  current  maintenance  data  collection 
system  is  not  adequate  for  measurement  of  the  three  parameters  of  inter¬ 
est  —  FFI,  FFD,  and  FFA  —  largely  because  there  is  no  current  mechanism 
for  identifying  those  fault  detection  or  isolations  due  to  "defined 
means,"  such  as  pilot  check  lists,  troubleshooting  check  lists,  or  built- 
in  test  equipment..  Without  such  an  indicator,  we  cannot  identify  the 

terras  MA  .  CND  ,  or  FI  in  the  equations  for  FFI  or  FFD.  However, 
s  s  s 
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If  B  is  assumed  co  be  close  to  1,  we  can  measure  FCND,  an  estimator  of  FFA, 
as  discussed  in  Section  6.2.  There  is  some  rationale  for  doing  this,  since 
any  CND,  whether  a  false  alarm  or  not,  has  a  detrimental  effect  on  opera¬ 
tional  readiness.  Design  effort  to  reduce  CNDs  would  be  ultimately  bene¬ 
ficial.  In  addition,  PCND  represents  an  upper  limit  on  FFA. 

We  will  show  In  this  section  that  there  appears  to  be  some  relation¬ 
ship  between  this  FCND  and  at  least  one  design  parameter,  parallelism. 

6.4.1  The  Data  Set 

Chapter  Four  addressed  potential  data  sources.  In  anticipation  of 
studying  the  potential  relationships  of  the  three  parameters  to  design 
data,  we  obtained  a  six-month  history  of  a  tactical  fighter  wing  mainte¬ 
nance  activities  (AFTO  349  summaries)  on  a  limited  set  of  avionics.  We 
were  able  to  obtain  technical  manuals  for  three  of  these  LRUs.  We  further 
obtained  pilot  debrief  reports  and  AFTO  349  summaries  for  a  set  of  31  LRUs, 
covering  three  months  at  four  bases  from  a  special  test  being  conducted  on 
a  second  tactical  aircraft.  This  set  was  culled  to  nine  LRUs  by  requiring 
that  at  least  30  records  (maintenance  actions)  be  present  for  each  LRU  in 
the  final  set.  We  then  obtained  some  abbreviated  technical  data  In  the 
form  of  block  diagrams  and  SRU  counts  for  each  of  these  LRUs. 

The  LRUs,  associated  maintenance  data,  and  available  technical 
(design)  data  are  shown  In  Table  7.  We  used  the  number  of  connectors  as  a 
rough  estimator  of  the  number  of  Inputs  for  a  given  LRU.  The  resulting 
parameter  is  called  “connector  dependency." 

6.4.2  FFA  Relationships 

The  last  column  of  Table  7  summarizes  the  PCND  derived  from  the  main¬ 
tenance  data.  We  conducted  a  series  of  regressions  of  PCND  versus  degree 
of  parallelism  and  versus  connector  dependency,  shown  in  Plgures  6  through 
9.  Connector  dependency  displayed  very  poor  correlation  with  PCND,  but 
an  encouraging  correlation  between  PCND  and  degree  of  parallelism  was  dis¬ 
covered.  as  can  be  seen  In  Plgures  10  through  13.  The  exponential  form 
had  the  highest  correlation  with  the  data,  and  It  Is  probably  the  most 
defendable  functional  form  of  those  attempted  (linear,  logarithmic,  expo¬ 
nential.  and  power).  Clearly.  PCND  must  have  an  asymptotic  upper  limit, 
and  that  limit  should  be  lower  than  1,  so  that  a  linear  form  should  be 
rejected.  Furthermore.  PCND  cannot  tend  to  a  negative  number,  so  that  the 
logarithmic  form  should  also  be  rejected. 


If  we  accept  the  hypothesis  that  PCND  has  a  non-zero  lower  limit,  we 
can  Imagine  that  PCND  might  grow  as  the  degree  of  parallelism  (connective 
complexity)  increases.  When  this  complexity  begins  to  cause  PCNDs 
approaching  50  percent,  design  emphasis  Is  placed  on  reducing  that  growth 
In  PCND.  Hence,  downward  pressure  Is  exerted  on  the  curve,  and  It  tends 
to  flatten.  One  could  argue  that  this  emphasis  is  placed  after  the  design 
becomes  rather  Inflexible,  and  hence  "band-aid"  fixes  must  be  applied. 


TABLE  7.  LRU  MAINTENANCE  AND  DESIGN  DATA 
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FIGURE  10.  LINEAR  FIT  OF  FCND  VERSUS  CONNECTOR  DEPENDENCY 
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FIGURE  11.  LOGARITHMIC  FIT  OF  FCND  VERSUS  CONNECTOR  DEPENDENCY 
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WE  12.  EXPONENTIAL  FIT  OF  FCND  VERSUS  CONNECTOR  DEPENDENCY 


FIGURE  13.  POWER  FIT  OF  FCND  VERSUS  CONNECTOR  DEPENDENCY 


Such  fixes  tend  not  to  address  the  false-alarra  problem,  so  the  ultimate 
design  may  still  have  higher  FCND  than  would  be  possible  with  good 
design-for-testabl 11 ty . 


Consequently,  a  functional  form  that  Is  asymptotic  at  both  small  and 
large  degrees  of  parallelism  (l.e..  S-shaped)  Is  a  more  defendable  func¬ 
tional  form  than  a  pure  exponential  or  power  expression.  However,  because 
the  data  set  on  which  these  regressions  are  made  is  meager,  it  Is  not  par¬ 
ticularly  beneficial  to  carry  this  analysis  much  further.  Clearly  there 
appears  to  be  some  physically  justifiable  functional  form,  and  the  avail¬ 
able  data  display  a  fair  degree  of  agreement  with  that  form.  Ve  caution 
the  reader  that  these  regressions  only  indicate  some  hope  of  identifying  a 
functional  form  for  FFA  when  measured  with  a  small  data  set,  and  in  no  way 
represent  relationships  that  could  be  used  as  FFA  predictors  during  the 
design  process. 

In  addition  to  the  usual  statistical  scatter  in  the  dependent  vari¬ 
able  (FCND),  there  is  the  same  uncertainty  in  the  accuracy  of  the  indepen¬ 
dent  variable,  because  not  all  of  the  technical  data  employed  to  determine 
this  measure  were  of  the  same  level  of  detail.  For  example,  some  equip¬ 
ment  Technical  Orders  to  which  we  had  access  show  "functional  flow"  dia¬ 
grams,  while  others  were  more  closely  related  to  wiring  diagrams,  which 
might  yield  a  different  degree  of  parallelism. 

Other  important  factors  associated  with  the  potential  relationship 
between  FCND  and  design  parameters  are  measurability  and  statlonarlty. 

The  flow  model  showed  that  data  must  be  taken  over  a  long  enough  period  so 
that  any  natural  periodicity  can  be  averaged  out.  Figures  14  through  16 
show  the  time  variations  in  FCND  for  three  LRUs  over  a  six-month  period. 
The  variations  shown  in  these  figures  are  caused  by  a  combination  of  time- 
cyclic  variations  in  FCND  and  sampling  variations.  Both  are  handled  by 
using  a  large  number  of  data  points.  In  the  cyclic  variation  case  the 
period  covered  by  the  data  samples  must  be  long  enough  to  cover  a  com¬ 
plete  cycle.  Sampling  variation  can  be  reduced  by  collecting  a  large  num¬ 
ber  of  data  points  representing  each  period  in  the  cycle.  For  example,  if 
a  complete  cycle  is  seven  months,  the  "average"  FCND  should  be  taken  over 
an  integral  number  of  seven-month  cycles.  Each  month  of  that  cycle  should 
consist  of  a  large  number  of  data  points  in  order  to  reduce  sampling 
variation.  The  quantity  of  data  necessary  to  meet  these  criteria  will  be 
investigated  in  the  second  phase  of  this  study. 

6.4.3  Summary  of  the  Examined  Relationship 

We  have  shown  that  there  is  some  justifiable  relationship  between 
FCND  (FFA  with  B  *  1.0)  and  a  design  parameter  for  the  limited  set  of 
avionics  shown  in  Table  6.  The  standard  maintenance  data  collection  sys¬ 
tem  currently  employed  by  the  Air  Force  is  not  structured  to  collect  the 
information  needed  to  obtain  estimators  for  FFD  or  FFI.  or  to  determine 
false-alarra  rate  or  fraction  of  false  alarms.  Without  these,  it  is  chal¬ 
lenging  to  investigate  the  existence  of  functional  forms  relating  design 
parameters  to  field  maintenance  and  testability. 


MONTHS 


MONTHS 


FIGURE  16.  VARIATION  OF  FCND  WITH  TIME  FOR 
AIRPLANE  B,  LRU  12 


The  missing  connection  Is  some  method  for  Identifying  instances  in 
which  "defined  means"  triggered  or  closed  a  maintenance  action,  and  for 
estimating  the  empirical  coefficients  B  and  B  . 


6.5  ANOMALY  CHECK 

It  is  not  possible  to  evaluate  this  last  element  of  the  predictor 
feasibility  decision  process  at  this  time,  since  measures  of  FFA,  FFD,  and 
FFI  are  not  available  to  evaluate.  We  can  observe  that  at  least  one 
researcher  has  noted  large  variations  in  CND  from  base  to  base  on  the  same 
equipment,  and  we  have  noted  a  potential  for  CNDs  to  be  polluted  by  manage 
ment  emphasis.  Given  the  critical  role  of  CND  in  the  definition  of  FFA, 
FFD.  and  FFI,  there  is  certainly  a  potential  for  "unexplained  anomalies." 
It  is.  however,  too  early  to  draw  conclusions  as  to  the  impact  of  anom¬ 
alies  on  any  prediction  technique  that  may  be  developed. 


6.6  SUMMARY  OF  PREDICTOR  FEASIBILITY 

We  employed  the  feasibility  decision  model  shown  in  Figure  4  on  each 
of  the  three  testability  parameters  under  analysis  --  FFA.  FFD,  FFI.  All 
three  parameters  were  found  to  be  potentially  practical  but  not  practical 


now,  because  they  cannot  be  measured  with  the  current  maintenance  report' 
Ing  system.  The  resulting  decision,  based  upon  alternative  approaches  to 
measurement,  translates  to  a  feasible  decision. 

To  Identify  other  potential  roadblocks  to  developing  predictors  for 
these  testability  parameters,  we  continued  through  the  next  two  tiers  of 
the  decision  model,  we  concluded  that  there  are  existing  design  param¬ 
eters  that  should  have  a  bearing  on  FPI,  FFD,  and  FFA.  we  showed  that  a 
modified  FFA  (FCND),  which  is  really  a  CND  fraction,  can  be  measured  and 
that  approximately  50  percent  of  the  variance  In  CND  rate  for  a  small  set 
of  LRUs  can  be  attributed  to  one  design  parameter,  the  ratio  of  the  number 
of  parallel  paths  In  a  system  to  the  number  of  subelements  In  that  system. 

The  results  of  the  Phase  I  study  are  guardedly  encouraging.  The 
development  of  the  B  parameter,  through  any  one  of  a  number  of  techniques, 
will  allow  a  first -order  approximation  to  field  FFA.  Alternative 
approaches  must  be  sought  for  the  development  of  FFI  and  FFD.  These 
approaches  would  include  exploitation  operational  tests,  M-demo  results, 
and  the  use  of  available  testability  models. 


CHAPTER  SEVEN 


SUMMARY,  CONCLUSIONS,  AND  RECOMMENDATIONS 


7.1  SUMMARY  AND  CONCLUSIONS 

The  Phase  I  efforts  led  to  the  following  conclusions: 

-  Definitions  of  key  organizational-level  testability  parameters 
(FFD.  FFI.  and  FFA)  that  are  consistent  with  existing  documenta¬ 
tion  and  measurable  at  the  organization  level  have  been  developed. 

-  Sufficient  mathematical  frameworks  have  been  derived  to  permit  the 
consistent  measurement  of  these  parameters  for  the  development  of 
prediction  based  on  design  attributes. 

-  The  Air  Force  Technical  Order  (AFTO)  Maintenance  Data  Collection 
System  does  not  currently  report  all  of  those  items  necessary  to 
measure  the  aforementioned  parameters. 

—  The  proper  measurement  of  FFD  will  require  a  report  containing 
the  genesis  of  the  maintenance  activity,  with  particular  empha 
sis  on  whether  or  not  the  normal  system  maintenance  procedures 
were  responsible  for  the  maintenance  action. 

—  The  proper  measurement  of  FFI  will  require  a  report  containing 
the  basis  of  the  resolution  of  a  maintenance  activity  —  Spe¬ 
cifically,  whether  or  not  the  normal  system  maintenance  proce¬ 
dures  were  sufficient  for  resolution  of  the  maintenance 
activity. 

—  The  proper  measurement  of  FFA  will  require  some  means  of  sepa¬ 
rating  "cannot  duplicate"  events  of  real  failures  from  "cannot 
duplicate"  events  of  nonfailures,  with  emphasis  on  the 
tracking  of  maintenance  history  for  some  "bad  actors." 

-  The  insufficient  measurability  of  the  parameters  listed  above  has 
restricted  the  feasibility  work  and  limited  it  primarily  to  inves¬ 
tigation  of  the  FFA  parameter. 

-  Sufficient  evidence  exists  to  support  the  conjecture  that  the 
building  of  a  prediction  technique  is  feasible  given  measurable 
field  parameters. 


-  Algorithmic  techniques  for  combining  subsystem  data  Into  system 
estimators  have  been  developed  but  remain  unverified. 

-  Techniques  for  the  development  of  several  of  the  empirical  coeffi¬ 
cients  (0.  6.  Y)  need  further  development. 

-  There  are  other  bases  for  the  computation  of  parameters  related  to 
FFD.  FFI,  and  FFA  that  are  Independent  of  field  data. 

-  The  lack  of  measurability  of  field  FFD.  FFI.  and  FFA  precludes 
field  verification  of  any  predictor  model  at  this  time. 

-  There  are  fully  automated  maintenance  data  collection  systems  that 
could  be  modified  to  make  the  analysis  of  large  data  samples 
practical. 

-  All  of  the  elements  necessary  to  achieve  prediction  techniques 
either  exist  or  are  obtainable. 


7.2  RECOMMENDATIONS 

The  following  actions  are  recommended: 

-  Proceed  with  the  development  of  a  first-order  prediction  technique 
for  the  fraction  of  false  alarms  (FFA)  at  the  organizational  level 
based  on  empirically  derived  coefficients. 

-  Proceed  with  the  development  of  prediction  estimates  of  fraction 
of  faults  detected  (FFD)  based  on  detail  design  analysis,  opera¬ 
tional  evaluation  data,  and  maintenance  demonstration  data. 

-  Proceed  with  the  development  of  prediction  estimates  of  fraction 
of  faults  isolated  (FFI)  based  on  detail  design  analysis,  opera¬ 
tional  evaluation  data,  maintenance  demonstration  data,  and 
testability  modeling. 


MAINTENANCE  PROCEDURE  QUESTIONNAIRE 


A  better  understanding  of  our  goals  will  be  achieved  if  the  entire  question¬ 
naire  is  read  before  proceeding  with  answers.  The  following  terns  are  used  in  the 
questionnaire: 

-  Normal  Maintenance.  Normal  pre-  and  post-flight  checks ,  and  the  use  of  BIT 
and  hardware-specific  ATE,  as  well  as  semiautomatic  or  manual  troubleshoot¬ 
ing  procedures  outlined  in  the  test  procedures  manual.  It  does  not  Include 
any  nonprescribed  troubleshooting  or  ‘shotgun*  maintenance  procedures. 

-  cannot  Duplicate  ( CMP ) .  The  troubleshooting  procedures  indicate  that  the 
system  is  fault-free  (no-fault-found);  i.e. ,  where  there  has  been  an  indica¬ 
tion  of  failure,  either  pilot  report  or  BIT. 

-  Fault  Isolation.  A  sufficient  degree  of  information  is  obtained  to  identify 
all  failed  replaceable  units. 

-  Complex  System.  A  system  consisting  of  multiple  replaceable  units  that  are 
interconnected  in  such  a  way  as  to  make  fault  isolation  difficult. 


Check  here  if  you  would  like  a  copy  of  the  survey  results  and  study  findings. 


Ql.  We  are  dealing  with  the  analysis  of  organizational  maintenance  of  complex  elec¬ 
tronic  systems.  Approximately  how  many  such  systems  are  maintained  by  your 
facility? 

(e.g.,  AN/yyy-XXX,  Radar  System,  Flight  Control  System,  Inertial  Navigation 
System,  EW  System  —  not  individual  replaceable  assemblies) 


A.  Estimated  percentages  of: 

_ Contractor-maintained  _______  Other  (explain) 


_________  Military  personnel-maintained 

B.  Comments  _ 


Q2 .  What  reporting  systems  do  you  receive  from  and  supply  information  to?  (e.g.. 
Maintenance  Data  Collection  System  (MDC),  AFM-66-1,  Naval  Aviation  Logistics 
Data  Analysis  System  (NALOA),  Maintenance  Material  Management)  Check  all  that 
apply. 

Data  System  Receive  Info  From  Supply  Info  To 


Comments 


1 


Q3.  Do  you  maintain  local  maintenance  action  filaa  for  complex  alactronic  systems? 
Yaa  Wo 

A.  It  yaa,  how  long  a  calendar  period  la  represented  by  your  filaa? 


B.  If  yaa,  ara  you  willing  to  hava  these  filaa  examined  for  raaearch  purposes? 

_ Yea  Wo 

C.  Do  your  records  contain  inforaation  on: 

_____  Repair  times?  _____  Fault  isolation  times? 

D.  Do  you  keep  maintenance  history  files  on  particularly  bad  problems  (subse¬ 
quent  repairs  from  repeated  gripes)? 

_ Yes  _ Wo 

E.  Are  multiple  failure  replacements  handled  by  one  report  or  by  multiple 
reports? 

_ One  __  Multiple 

F .  How?  _ _ 

Comments  on  local  maintenance  action  files  _ 


Q4 .  Do  you  differentiate  in  reporting  between  maintenance  action  triggered  by  oper¬ 
ator  complaints  and  maintenance  action  triggered  by  routine  maintenance? 

_ Yes  _ Wo 

A.  If  yea,  how  do  they  differ  in  reporting?  ________________________ 


B.  Do  you  report  which  stage  of  routine  maintenance  triggered  a  maintenance 
action  (e.g.,  pre-flight,  in-flight  BIT,  post-flight,  calendar  checks)? 

_ Yes  _____  No 

C.  when  routine  maintenance  gives  a  Wo-Fault-Found  (WFF)  or  Cannot  Duplicate 
( CWD ) ,  do  you  differentiate  between  suspected  intermittent  and  simply  un¬ 
verified  problems? 

_ Yea  _ Wo 


D.  Bow? 


E.  Do  you  record  the  aenner  in  which  final  replaceable  units  are  isolated 
(e.g. ,  BIT,  ATE,  semiautomatic,  manual  by  the  book,  nonprescribed 
procedures) . 

_____  *•«  _ No 

F.  Bow/Comments  _ 


Q5.  Do  you  record  instances  in  which  the  local  technician  is  unable  to  isolate  a 
problem  uncovered  during  normal  maintenance? 

_ Yes  _ No 

A.  How/Comments  _ 


B.  How  are  these  instances  (of  Q5)  handled  in  general  (e.g.,  multiple  replace¬ 
ments,  bring  in  more  talent,  send  unit  under  test  out)? 


Comments 


The  following  variables  are  used  in  trial  definitions.  Please  consider  each 
carefully;  choose  the  most  appropriate  or  the  most  in  line  with  your  view.  You  may 
provide  an  alternative  definition  or  a  critique  of  the  listed  definitions. 


Q6.  Number  of  false  alarms,  based  on: 

A.  ______  The  total  number  of  Cannot  Duplicates  (CNDs)  or  No-Fault-Found  (NFF). 

B.  _  The  total  number  of  CNDs,  excluding  operator-reported  faults  that  re¬ 

sult  in  a  CND. 

C.  _____  The  total  number  of  CNDs  as  in  B,  also  excluding  those  tagged  as  sus¬ 

pected  intermittents  by  maintenance. 


Alternative  Definition  or  Comments 


Q7.  Number  of  faults  detected,  based  on: 


A.  _  The  total  number  of  maintenance  actiona  triggered  by  normal  mainte¬ 

nance  that  do  not  result  in  a  CND. 

B.  _  The  total  number  of  failure  modes  triggered  by  normal  maintenance  that 

do  not  result  in  a  CND. 

C.  _  The  number  of  faults  detected  and  isolated  by  normal  maintenance. 

Alternative  Definition  or  Comments  _ _ 


Q8.  Number  of  faults  isolated,  based  on: 

A.  _  The  total  number  of  faults  isolated  to  a  replaceable  unit  using  only 

defined  maintenance  procedures. 

B.  _  (A)  plus  standard  troubleshooting  techniques. 

C.  _____  (A)  minus  the  operator  triggered  maintenance  actions. 

D.  _  all  of  (AJ  that  are  by  BIT/ATE  only 

Alternative  Definition  or  Comments:  _ 


Q9 .  Philosophy.  Please  provide  us  with  insights  you  might  have  into  keys  to  im¬ 
provement  and  important  factors  that  determine  the  following:  false  alarms, 
fault  detections,  fault  isolation,  or  others. 


Q10 .  Can  you  recommend  other  sources  of  Information? 
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APPENDIX  B 


MATHEMATICAL  REPRESENTATIONS  OP  ORGANIZATIONAL  MAINTENANCE 


1.  MATHEMATICAL  MODELING  APPROACH 

Mathematical  modeling  was  undertaken  to  provide  measurement  algo¬ 
rithms  that  were  consistent  with  the  derived  definitions.  We  started  with 
the  modified  state  model,  which  Is  Intended  to  relate  the  organizational 
testability  parameters  to  a  search  for  the  system  state.  This  represents 
the  heart  of  the  maintenance  problem  In  that  there  is  an  Indication  of  a 
problem  and  action  must  be  taken  to  find  out  whether  a  real  failure  is 
present  and  where  it  Is  located.  As  this  model  was  being  developed.  It 
became  apparent  that  conflicts  In  definitions  were  surfacing.  For  exam¬ 
ple.  It  was  not  clear  whether  a  “cannot  duplicate"  (CND)  event  and  a  fault- 
isolation  event  were  mutually  exclusive.  To  resolve  these  problems,  a 
second  model,  based  on  membership  in  sets,  was  developed.  The  primary 
tool  was  the  Venn  diagram.  In  which  both  mutual  exclusivity  and  coincident 
properties  are  explicit.  This  set  model  led  to  a  clear  and  concise  set  of 
definitions  that  were  mathematically  precise,  as  well  as  an  algorithm  set 
that  could  be  used  to  verify  the  other  models.  The  state  model  was  then 
reworked  on  the  basis  of  definitions  generated  by  the  set  theory  model 
with  most  of  the  conflicts  resolved. 

A  third  model,  based  on  the  flow  of  maintenance  events,  was  developed 
and  was  pursued  concurrently  with  the  other  two  models.  This  model  was  to 
solve  two  of  the  problems  being  faced.  The  first  problem  was  relating 
maintenance  actions  to  readiness.  While  a  preliminary  connective  had  been 
established  with  the  modified  state  model.  It  was  less  than  satisfactory. 
The  flow  model  would,  by  tracing  events  through  the  mission/maintenance 
cycle,  provide  a  direct  tie-in.  The  second  problem  was  more  basic:  There 
was  no  direct  way  to  relate  our  mathematical  approach  to  the  maintenance 
personnel.  The  first  two  models  were  too  "mathematical."  The  flow  model 
was  readily  analyzed  by  maintenance  personnel  of  SAC,  TAC,  and  MAC,  and 
underwent  major  revisions  based  on  discussions  with  those  personnel.  As  a 
clearer  picture  of  the  organizational-level  maintenance  process  evolved, 
modifications  were  made  to  both  of  the  other  models.  Finally,  a  flow 
model  evolved  that  was  satisfactory  to  both  organizational  level  mainte¬ 
nance  personnel  and  the  mathematicians.  Symbols  used  in  this  section  are 
defined  in  the  Acronyms  and  Symbols  section  of  this  report. 

A  detailed  review  of  each  of  the  models  is  presented  in  this  appen¬ 
dix.  and  the  final  form  of  the  definitions  is  reviewed. 


2.  THE  SET  THEORY  REPRESENTATION 


In  order  to  develop  a  consistent  set  of  definitions  by  which  to  pro¬ 
ceed,  a  set  theory  representation  of  failure  and  maintenance  events  as 
shown  in  Figure  B-l  was  developed.  It  Is  through  set  membership  that  a 
consistent  set  of  definitions  will  emerge.  Each  of  the  set  designations 
will  be  examined  separately.  Two  basic  assumptions  are  made: 

1.  Any  fault  Indication  that  does  not  result  in  a  maintenance  action 
is  a  nonrelevant  event.  Under  most  definitions  of  system  behav¬ 
ior  these  would  be  called  false  alarms  that  are  recognized  but 
Ignored;  they  are  totally  unmeasurable  and  have  little  Impact  on 
the  maintenance  system  and  are  therefore  considered  nonrelevant. 

2.  Failures  that  are  not  detected  by  any  means  are  nonrelevant. 

This  last  point  will  be  discussed  in  detail  later  In  this 
appendix. 

These  assumptions  imply  that  latent  failures  (l.e.,  failures  that  are 
present  In  the  system  but  were  not  discovered  because  the  requisite  sub¬ 
system  was  not  exercised)  will  not  be  dealt  with  until  such  time  as  they 
are  discovered  and  trigger  a  maintenance  action. 

Section  2.1  through  2.7  are  discussions  of  the  primary  sets  shown  In 
Figure  B-l.  Each  of  these  sections  includes  a  Venn  diagram,  algebraic 
terms,  and  related  maintenance  terms,  followed  by  a  brief  discussion  of 
the  set. 


2.1  Universe  of  System  Configurations1 


(Maintenance  Term: 
None) 


This  set  represents  the  mapping  of  all  systems  for  which  the  organi¬ 
zational  maintenance  unit  has  responsibility.  It  includes  failed  and  non- 
falled  units,  those  undergoing  maintenance  or  performing  a  mission,  and 
those  simply  available  for  a  mission.  The  universe  Is  the  departure  point 
for  further  calculations  and  will  Include  the  definition  and  breakdown  of 
systems  and  subsystems. 


xln  this  appendix,  a  configuration  represents  a  system  state  consisting 
of  combinations  of  equipment  states,  failure  indications,  and  maintenance 
events  (e.g.,  available,  in  repair,  undergoing  checks). 
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(Maintenance  Term: 

F  -  Failures  or  Faults) 


This  set  represents  the  subset  of  the  universe  that  contains  fail¬ 
ures.  Its  members  are  not  directly  measurable  but  may  be  estimable  by 
FMECA  and  RAM  analysis.  This  set  points  out  the  real  problem  of  detec¬ 
tion.  Obviously,  only  failures  that  are  detected,  by  some  means,  can  be 
measured.  In  fact,  it  can  be  conjectured  that  a  failure  that  is  not 
detected  by  some  means  in  the  long  run  does  not  matter,  because  it  must 
have  an  imperceptible  Impact  on  the  mission.  An  example  was  pointed  out 
in  our  discussions  with  MAC  personnel:  a  wiring  problem  on  one  of  the  C-5 
intercom  systems  was  present  in  an  aircraft  for  many  missions  because  the 
failure  manifested  Itself  only  when  two  of  the  crew  were  on  different 
intercom  channels.  It  turned  out  to  be  an  undetected  failure  for  an 
unknown  number  of  missions,  because  it  did  not  affect  those  missions.  It 
finally  became  a  crew-  or  operator-reported  maintenance  discrepancy  when 
an  attempt  to  use  the  Intercom  in  this  mode  was  made  on  one  mission.  The 
only  measurable  event  was  the  operator  report  and  subsequent  maintenance 
action. 

2.3  Set  of  Failure  Indications 


(Maintenance  Term: 

MA  -  Maintenance  Actions) 


This  set  represents  the  subset  of  the  universe  that  results  in  a 
failure  indication  and  subsequent  maintenance  action.  It  is  the  universe 
of.  and  is  measurable  by,  maintenance  actions.  It  will  Include  some  fail¬ 
ures  and  some  nonfailures.  This  set  is  called  the  universe  of  maintenance 
actions  because  anything  that  can  be  measured  at  the  organizational  level 
is  Included  in  this  space.  Outside  this  set  we  can  only  estimate  data  on 
the  basis  of  known  or  conjectured  system  characteristics.  Inside  this  set 
we  may  be  able  to  develop  hard  evidence,  based  on  maintenance  reporting, 
for  some  of  the  set  attributes. 
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2.4  Set  of  Isolation  and  Repair  Events 


(Maintenance  Term: 

FI  -  Fault  Isolation) 
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This  set  represents  the  collection  of  maintenance  actions  that  result 
in  isolation  and  repair.  Members  of  the  set  are  inherently  measurable, 
although  care  must  be  taken  to  remove  biases  in  the  maintenance  reporting 
system.  It  has  been  observed  that  the  use  of  CND  rate  for  a  "grading" 
criterion  in  the  shops  can  lead  to  reporting  a  CND  as  a  recalibration  to 
avoid  the  high  CND  rate.  This  practice  could  misrepresent  a  CND  as  an 
isolation  and  repair  event.  This  set  is  a  subset  of  C. 


2.5  Set  of  Isolation  and  Repair  Events  Not  bv  Normal  Svstem  Maintenance1 


(Maintenance  Term: 


FIq  -  Nonsystem  Fault  Isolation) 


This  set  represents  Isolation  and  repair  events  that  are  accomplished 
outside  the  provided  maintenance  structure  for  the  specific  eguipment. 

The  provided  structure  in  these  cases  fails  to  give  the  information 
necessary  for  isolation  and  repair;  and  the  experience,  training,  and 
intuition  of  maintenance  personnel  are  called  upon  to  make  the  final 
determination.  The  more  fully  automated  the  system  is  today,  the  less 
likely  a  distinction  will  be  measurable  between  sets  D  and  E.  The  older, 
manually  driven  reporting  systems  such  as  SAC  B-52  maintenance  will 


xNormal  system  maintenance  -  Techniques  that  are  specified  as  standard 
operating  procedures  for  use  of  BIT,  ATE.  semiautomatic,  or  documented 
manual  detection  and  troubleshooting  for  a  given  system  under  test.  They 
include  regular  calendar  checks  and  normal  "go”  checks.  This  is  sometimes 
called  "defined  means"  (RADC  Testability  Notebook). 


ii 


Include  comments  that  might  provide  the  information  necessary  to  define 
this  set.  Such  older  system  reports  of  AFTO  349  data  are  not  amenable  to 
computer  processing,  because  the  comment  fields  require  manual  sorting. 

In  general,  however,  the  set  membership  Is  measurable  and  could  be 
reported.  This  set  is  a  subset  of  C  and  0. 

2.6  Set  of  Failure  Indications  Not  bv  Normal  System  Maintenance 


(Maintenance  Term: 

MA  -  Non-NSM  Maintenance  Action) 
o 


This  set  represents  the  failure  indications  that  occur  outside  normal 
system  maintenance.  These  are  typical  pilot-  or  crew-reported  malfunc¬ 
tions  that  BIT  or  other  normal  system  maintenance  does  not  also  report. 

The  failure  indication  may  also  be  the  result  of  a  maintenance  analysis. 
This  set  is  a  subset  of  C.  Current  systems  measure  this  set  only  in  the 
AFTO  349  comments,  but  this  set  membership  could  be  reported  separately. 

2.7  Cannot  Duplicate  Events 


(Maintenance  Term: 

CND  -  Cannot  Duplicate) 


This  set  represents  the  failure  indication  events  that  result  in  a 
maintenance  determination  of  "cannot  duplicate"  (CND).  There  are  many 
reasons  for  the  CND.  only  some  of  which  are  related  to  false  alarms.  G  is 
a  subset  of  C  and  is  exclusive  of  D  and  E.  It  is  not  only  measurable  but 
reported  under  AFTO  349  maintenance  reporting  schemes.  Care  must  be  taken 
to  avoid  biases  such  as  those  recounted  under  Section  2.4. 


The  set  theory  representation  can  now  lead  to  a  more  precise  defini¬ 
tion  of  design  goals.  Ultimately,  It  Is  desirable  for  a  design  to  yield 
the  congruency  of  B,  C,  and  D,  and  to  force  sets  E,  F,  and  G  to  the  null 
set.  More  generally,  we  would  like  the  normal  system  maintenance  to 
detect  and  Isolate  all  faults  without  false  alarms,  CNDs,  or  maintenance 
technician  Intervention. 

2.8  Key  Parameter  Definitions 

The  definitions  that  follow  are  keyed  to  the  consistency  of  the  set 
theory  model  but  are  left  In  generic  terms  for  use  with  the  other  models 
that  are  to  be  applied  to  these  definitions.  Note  that  the  terras  “proper" 
and  "improper"  are  for  the  purpose  of  partitioning  and  do  not  imply  "cor¬ 
rect"  or  "incorrect";  "optimal"  or  "suboptiraal" ;  or  any  other  connota¬ 
tion.  For  example,  while  a  centralized  BITE  subsystem  will  correctly 
detect  faults  In  other  subsystems,  these  detections  will  still  be  Improper 
by  these  definitions,  We  will  first  break  these  down  for  a  system  as  a 
whole. 

System-level  definitions  are  as  follows: 

-  Fault  detection  -  Normal  system  maintenance  Indicates  that  the 
system  Is  not  functioning  properly,  and  this  indication  Is  the 
result  of  a  real  fault  within  the  system. 

-  Fault  isolation  -  NSM  identifies  all  failed  units  within  the  sys¬ 
tem.  An  attempted  isolation  can  have  any  of  the  following  results: 

—  Proper  fault  Isolation  -  Only  and  all  failed  units  are 
Isolated. 

-  Improper  fault  Isolation  -  All  but  not  only  failed  units  are 
Isolated. 

—  No  fault  isolation  —  Other  combinations  that  occur,  including 
only  but  not  all  failed  units. 

-  False  alarm  -  There  is  indication  of  a  failure  in  the  system  where 
none  exists.  False-alarra  rate  (FAR)  is  the  sum  of  false  alarms 
over  a  general  time  period  divided  by  that  time  period. 

Related  definitions  are  provided  in  the  Glossary. 

The  system  definitions  must  be  consistent  also  in  a  hierarchical 
sense  as  subsystems  are  built  up  Into  systems.  The  boundary  between  sys¬ 
tem  and  subsystem  must  be  defined  in  advance. 
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Sections  2.10  through  2.28  are  discussions  of  the  secondary  sets. 
Each  of  these  sections  includes  a  Venn  diagram,  algebraic  terms,  and 
related  maintenance  terms,  followed  by  a  brief  discussion  of  the  set. 

2.10  Undetected  Failures 


The  undetected  failure  (H)  is  represented  by  the  intersection  (n)  of 
the  failure  set  (B)  and  the  no-f ai lure-indication  set  (not  C.  or  com¬ 
plement  of  C).  These  are  truly  unmeasurable.  However,  meeting  mission 
objectives  dictates  that  all  relevant  failures  will  be  detected  by  some 
means,  and  detection  should  reduce  to  a  question  of  what  means.  Some 
noted  exceptions  to  this  may  be  the  MAC  communications  problem  discussed 
earlier,  or  the  failure  of  a  backup  system  when  the  primary  system  is 
fully  functional.  By  restricting  the  analysis  to  relevant  failures,  we 
are  assuming  that  H  goes  to 'the  null  set  <H  -*  null  set)  and  B  becomes  a 
subset  of  C  (every  member  of  B  is  also  a  member  of  C).  Set  H  will  include 
latent  defects  and  failures  until  such  time  as  they  are  discovered  and 
trigger  a  maintenance  action. 

2.11  Valid  Detections 


Valid  detections  are  the  events  that  reside  in  the  failure  set  and  in 
the  detection  set.  Under  the  relevant  failures  assumption,  B  is  a  subset 
of  C  and,  therefore,  B  n  C  «  B. 
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2.12  Valid  Detections  bv  Normal  System  Maintenance 


(Maintenance  Term: 

FD  -  Fault  Detected  by  NSM) 
s 


Under  the  relevant  failure  assumption.  It  becomes  important  to  delin¬ 
eate  which  of  the  failures  were  detected  by  NSM.  This  set  Is  asubset  of 
c.  If  H  -»  *  (B  n  c  *  B) .  then  B  n  (C  n  F)  s  (B  n  O  n  F  *  B  n  F, 
which  can  be  used  to  simplify  the  mathematics  somewhat. 


2.13  Valid  Detections  bv  Operators  Outside  NSM 


In  dealing  with  relevant  failures,  this  set  Is  the  B  complement  of  J. 
The  B  complement  of  J  means  that  K  U  J  *  B. 

Note:  K  u  J  3  (F  n  B)  u  (B  n  F)  from  definition  of  K 
and  J  with  relevant  failures 
»  (F  n  (BUB))n  (BUF) 

=  F  n  B  n  (Bu?) 

=  (F  n  F)  U  (B  n  B)  =  ♦  U  B  3  B 


This  represents  a  mismatch  between  the  system  failure  set  and  the  normal 
system  maintenance  detection  process. 


NITS/NRTS  and 


?r  Cannot  Duplicates 


L  =  C  n  (B  n  D) 


(Maintenance  Terms: 

NITS,  NRTS,  CND  -  Not  Isolatable 
This  Station,  Not  Repairable  This 
Station,  Cannot  Duplicate) 


This  set  represents  real  failures  that  cannot  be  verified  or  isolated 
and  repaired  by  the  organizational-level  maintenance  system  through  either 
defined  means  or  otherwise.  There  may  also  be  a  subset  that  Includes  war¬ 
ranty/guarantee  items  that  the  organizational-level  maintenance  is  not 
allowed  to  repair.  Under  relevant-failure  rules,  this  reduces  to  B  inter¬ 
secting  the  complement  of  D  (BOD). 

2.15  Isolation  bv  Normal  System  Maintenance  of  Real  Failures 


M  =  B  n  (D  n  E) 


(Maintenance  Term: 

FI  -  Fault  Isolation  by  NSM) 
s 


A  design  goal  is  to  make  all  failures  detectable  and  Isolatable  by 
normal  system  maintenance.  The  effectiveness  of  the  isolation  will  then 
represent  the  extent  to  which  this  set  is  congruent  with  the  set  of  all 
detected  failures.  The  set  then  is  given  by  the  relation  between  this  and 
set  I. 

2.16  False  Alarms 


N  =C  n  B 


(Maintenance  Term: 
FA  -  False  Alarms) 


This  set  represents  a  primary  testability  factor  and  represents,  by 
definition,  detection  of  nonfailures.  The  fact  that  the  diagram  shows 
that  so  many  subsets  impinge  upon  this  set  reflects  the  difficulty  that 
might  be  anticipated  in  measuring  it.  The  balance  of  the  set  definitions 
will  concern  false  alarm  in  one  way  or  another. 


2.17  Operator-Induced  False  Alarms 


O  =  F  n  B 


(Maintenance  Term: 

FA  -  False  Alarm  not  NSM) 


This  set  represents  the  false  alarms  triggered  by  operator-induced 
maintenance  actions.  Recognizable  false  alarms  that  do  not  result  in  a 
maintenance  action  will  not  affect  N  or  0. 


2.18  Cannot  Duplicate  Events  of  Real  Failures 


(Maintenance  Term: 
CND^  -  Improper  CND) 


This  set  represents  the  mismatch  between  the  detection  equipment/ 
environment  and  the  maintenance  equipment /environment.  In  this  set.  a 
detection  of  a  real  failure  cannot  be  duplicated  on  the  ground,  and  this 
may  lead  to  repeated  maintenance  actions  for  the  same  failure.  If  serial 
number  tracking  is  installed,  some  of  these  may  be  tagged  as  "bad  actors" 
and  will  move  from  CND  to  NRTS  (set  L'CflBnD). 


2.19 


>t  Duplicate  Events  of  False  Alarms 


Canno 


This  set  is  the  G  complement  of  Set  P  (P  u  Q  ■  G)  and  represents 
only  part  of  the  CMDs  and  part  of  the  false  alarms. 


2.20  Isolation  and  Repair  of  Nonfailures 


Although  this  set  Is  recognized  as  a  real  problem  of  maintenance 
systems.  It  Is  often  Ignored  (R  assumed  to  be  null  set)  because  of  the 
multiple  windows  It  must  pass  in  order  to  manifest  Itself.  In  a  typical 
sequence  a  fault  Is  indicated,  then  verified,  isolated,  and  repaired,  and 
the  repair  check-out  verifies  that  the  problem  has  been  rectified. 
Finally,  the  next  level  yields  an  RTOK,  indicating  that  It  was  a  false 
alarm.  This  should  be  distinguished  from  RTOK  due  to  improper  fault  iso¬ 
lation  by  either  "defined  means"  or  "shotgun"  approaches.  Under  "shotgun 
maintenance  this  set  may  be  significant.  An  example  of  this  shotgun 

approach  is  seen  In  Redball1  fixes.  The  degree  to  which  "shotgun" 
approaches  manifest  themselves  In  unnecessary  repairs  is  In  inverse  pro¬ 
portion  to  the  training  and  expertise  of  the  maintenance  crew.  A  further 

example  of  use  of  the  "shotgun"  approaches3  is  seen  where  the  normal 
system  maintenance  procedures  are  overly  complex  or  time-consuming.  In 
any  event,  the  problem  will  manifest  itself  as  a  high  RTOK  rate  from  the 
next  level  of  maintenance. 


^Redball  refers  to  a  last-ditch  effort  to  save  a  mission  when  the 
scheduled  aircraft  is  faulty.  TAC  and  SAC  call  this  "Redball,"  and  MAC 
calls  it  "Red  Streak."  It  has  also  been  referred  to  as  "Blue  Streak"  by 
SAC. 

aJack  Osborn,  CAD/CAM  Technology  Working  Group  Report  (IDA/OSD  R&M 
Study) .  August  1983. 


2.21  Valid  Detection  and  Isolation  Completely  Outside  Normal  System 


(Maintenance  Terms : 

FDq,  FI0  “  Fault  Detection  not 
NSM,  Fault  Isolation  not  NSM) 


2.24  Isolation  and  Repair  Events  of  Real  Faults  Entirely  bv  Normal  System 
Maintenance 

I 


One  design  goal  could  be  the  congruence  of  D,  B,  and  C,  meaning  that 
all  faults  are  detected  and  isolated  by  NSM.  This  set  measures  the  match 
between  the  NSM  within  the  maintenance  concept  and  the  system.  It  also 
Includes  RTOKs  due  to  ambiguous  Isolations. 


2.25  False-Alarm  Confusion  Area 


This  set  represents  elements  that  are  often  misrepresented  in  discus¬ 
sions  of  false  alarms.  It  Is  the  CND  of  real  failures  (which  are  not  false 
alarms)  and  the  unnecessary  repairs  (which  are  false  alarms).  Assignment 
within  these  two  sets  represent  the  most  difficult  measurement  problem. 

2.26  Retest -OK 


(Maintenance  Term: 
RTOK  -  Retest-OK) 


This  set  Is  actually  a  second-order  set  and  Is  outside  the  range  of 
the  maintenance  system.  It  Is  the  retest -OK  return  systems  from  the  next 
level  of  repair  and  Is  primarily  associated  with  Improper  fault  Isolation; 


The  set  theory  model  provides  a  vehicle  by  which  we  can  algo¬ 
rithmically  develop  the  desired  parameters.  Eac.i  measure  is  based  upon  a 
set  relationship  in  Figure  B-l.  We  must  first  define  a  population  opera¬ 
tor  Q.  The  population  operator  enumerates  the  membership  of  a  set  and 
can  be  given  as  follows: 

Q<Z)  -  liz  (1 

where 


4  •  1  iff  Z  €  Z 
z 


4*0  otherwise 


This  population  measure  may  be  directly  related  to  parameters  of 
interest.  For  example: 

(2(C)  -  XMA 

where  C  Is  the  universe  of  maintenance  actions.  Also. 

(2(D)  =  XFI 
(2(E)  »  IF IQ 

2.29.1  Fraction  of  Faults  Detected  (FFD) 


From  the  definitions: 

Q(J)  Q(B  n  (C  n  F)) 


B 

The  numerator  term  of  Equation  5  can  be  given  by 
a<j)  «2(Bn  (C  n  ?))  =  a<C)  -  a(c  nl)  -  q(F  n  b) 
a<j)  =  a(c)  -  a(N)  -  quo 
For  each  right-hand  term  of  Equation  7, 
a(C)  -  XMA 

a(K)  -  Xro0  -  2mao  -  XFAo 
so  that: 

(2(0  -  (2(K)  *  XMA  -  *  XMAg  +  SFAq 

Q(N)  -  XFA 
Therefore 

(2(J)  =  XMAe  +  X^  -  XFA 

S  V 

(2(J)  -  X®  -  XFA 
s  s 

and  the  denominator  term  of  Equation  5  is  given  by 
q(B)  -  a(C)  -  n(c  n  I)  +  g<b  n  c) 

(2(B)  -  (2(C)  -  Q(N)  +  (2(H) 


> 

-  *  -  *  »  *  - ' 
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(12) 
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(15) 

AVOWS* 

so  chat 


2(B)  -  XMA  -  IFK  +  XU 

Then 


(16) 


FFD 


2(J) 

2(B) 


IMAs  -  XFAS 
IMA  -  IFA  +  IV 


(17) 


and  finally,  since  undetected  failures  (undetected  by  any  means)  are  con¬ 
sidered  not  relevant  and  are  not  measurable,  then 


XU  -  0 


and 


FFD  * 


p*s :  lFAs 

XMA  -  XFA 


(18) 


(17a) 


2.29.2  Fraction  of  Faults  Isolated  (FFI) 

From  the  definitions: 

2(M)  2(B  n  (D  n  I)) 

FFI  “  2(B)  '  2(B) 

where  the  numerator  Is  given  by 
P(M)  -  XFIs 

and  the  denominator  Is  given  by  Equation  16  so  that 
IFIs 

FFI  ”  £MA  -  £FA  +  £U 


(19) 


(20) 


(21) 


Although  the  equation  is  computationally  accurate,  the  FFI  parameter 
can  be  misused  as  a  design  variable,  because  improper  fault  isolations  are 
as  valid  as  accurate  ones.  For  example,  a  "defined  means"  that  dictates 
wholesale  replacement  when  a  verified  fault  Is  discovered  will  achieve  a 
high  FFI.  but  a  very  high  RTOK  rate  at  the  next  level  of  maintenance. 

It  would,  of  course,  be  better  to  vise  the  proper  fault  isolations  -- 
that  Is.  fault  Isolations  that  Isolate  only  and  all  bad  elements,  or  fault 
Isolations  that  are  free  from  RTOK.  The  only  difficulty  is  in  measure¬ 
ment.  To  achieve  measurability  we  would  have  to  track  each  repair  by 
serial  number  and  maintenance  action  number  to  a  lower  level  of  mainte¬ 
nance  and  through  to  a  final  conclusion.  A  compromise  can  be  achieved  by 

penalizing  the  FI  for  RTOK  .  represented  algorithmically  by 
s  s 
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FFIp 


Q(M)  ~  0(Z) 
Q(B) 


IFI  -  JRTOK 

FFI  =  - - - — 

P  IMA  -  IFA  +  IU 


(19a) 

(21a) 


The  p  subscript  here  stands  for  performance.  This  does  achieve  a 
measure  that  drives  the  "defined  means"  toward  accurate  Isolation,  but  It 
may  have  range  and  domain  problems,  because  a  single  Isolation  could 
develop  multiple  RTOKs  and  the  overall  measure  could  be  0  or  less  than  0 
when  fault  Isolations  are  achievable.  We  recommend  that  both  FFI  and  RTOK 
together  be  considered  as  performance  measures. 


Either  Equation  21  or  Equation  21a  can  be  modified  for  nonrelevance  as 


*FIs 

FFI  £MA  -  I FA 


and 


FFIp 


IFIS  -  IRTQKS 
IMA  -  I  FA 


(21b) 
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2.29.3  Fraction  of  False  Alarms  (FFA) 
From  the  definitions: 


Q(N)  Q(C  O  B) 

PF1  m  1  m  ■■  ■ 

Q(C)  Q(C) 


(22) 


The  numerator  term  can  be  expressed  as 

Q(N>  -  0(G)  -  Q(G  flB)  ♦  Q(C  n  B)  -  J2(G  D  B)  (23) 

Q(M)  =  Q(G)  -  Q(P)  +  S2(N)  -  Q(Q)  (23a) 

where  each  member  on  the  right  side  of  Equation  23a  is  defined  as 

Q(G)  -  ICND  (24) 

Q(P)  -  ICW^  (25) 


Q(N)  -  Q(Q)  »  other  false  alarms  -  IRTOK  *  q(Y) 


(26) 


■ I  • 


Tne  preceding  discussion 
eratlon  of  system  data  from  s 
requires  an  accounting  of  the 
let  us  take  a  system  made  up 


Q(B  ) 

T 


QCBj)  - 


The  results  of  the  final  equation  set  are  summarized  below.  The 
fractions  of  faults  detected  is 

TMA„  -  (YCND  -  YCNDt  +  IRT0K„)H 


XU  +  IMA  -  (XCND  -  XCNDx  +  XRTOKu) 


(combined  Equations  17  and  27) 


[• 


or.  with  stated  relevancy  assumptions 
IMA,  -  <ICND  -  ICNDt>„ 


IMA  -  (ICND  -  ICNDj) 

The  fraction  of  faults  Isolated  Is 


IU  +  IMA  -  (ICND  -  ICNDt  +  IRTOK  ) 


IU  +  IMA  -  (ICND  -  ICNDj  +  lRTOKy) 
or.  with  stated  relevancy  assumptions: 


IMA  -  (ICND  -  ICNDt) 


P  IMA  -  (ICND  -  ICNDj) 

The  fraction  of  false  alarms  Is  simply  given  by  Equations  27  and  27a 
Finally,  the  false-alarra  rate  can  be  written  as: 

I FA  ICND  -  ICNDt  +  IRTOK 

FAR  -  —  *  - zr - - 

T  T 

(32) 

and. 

with  relevancy  assumptions: 

ICND  -  ICND 

FAR - ^ - - 

(32a 

2.31 

Measurability  and  the  Set  Theory  Representation 

The  set  theory  approach  will  require  the  resolution  of  each  mainte¬ 
nance  event  Into  its  membership  In  the  six  sets  (B  through  G) .  This  Is  a 
difficult  task  because  nonmaintenance  events  occur  In  B.  These  we  have 
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i 

I 

I 

I 

called  nonrelevant.  Even  under  the  relevancy  assumption.  B  becomes  the 
most  difficult  set  to  map.  In  many  Instances  this  will  require  "back¬ 
filling"  data.  That  Is.  the  membership  of  set  B  will  be  determined  at 
I  some  time  after  the  maintenance  event  by  examining  a  collection  of  sub- 

!  sequent  data. 

For  example,  CNDj  requires  set  membership  In  CND  (set  G)  and  real 

faults  (set  B) .  One  could,  through  serial  number  tracking,  resolve  "bad 
actors."  If  through  this  analysis  a  particular  system  Is  tagged  a  “bad 
|  actor,"  then  previous  CNDs  could  be  assumed  to  be  members  of  set  B  (real 

!  faults).  Other  backward  tracings  might  include  RTOK  (if  considered 

s 

necessary),  or  the  unnecessary  repairs.  RTOK^.  In  this  last  case,  at 

least  a  rationale  has  been  developed  for  ignoring  this  problem.  Other 
measurement  considerations  include  how  the  maintenance  action  came  about 
(with  or  without  NSM  -  set  C  or  set  F)  and  how  the  isolation  came  about 

(without  or  with  NSM  -  set  D  or  set  E).  These  latter  are  handled  only  in 

the  comments  section  of  the  normal  AFTO  349  reports. 

The  equation  set  30a,  31b,  27a,  and  32a  illustrates  the  reliance  of 
all  of  the  parameters  on  the  false-alarm  measure.  The  separation  of  CND 
into  its  component  parts  is  a  necessity  for  continuing  with  any  degree  of 
confidence,  our  interviews  with  SAC.  MAC.  and  TAC  have  indicated  that 
this  measurement  under  the  current  reporting  system  may  be  the  least  reli¬ 
able.  This  is  attributable  partly  to  the  dual  use  of  the  data  for  manage¬ 
ment  grading  and  maintenance  reporting,  leaving  the  measure  open  to  biases. 

2.32  Set  Theory  Approach  Summary 

The  set  theory  approach  has  delineated  the  requirements  of  a  “clean" 

measurement  system.  "Clean"  as  used  here  means  that  we  measure  only  and 

all  Information  necessary  in  the  system  to  define  our  parameters  of  inter¬ 
est.  Of  note  Is  that  the  necessary  measures  are  not  Included  In  any  one 
measuring  system  at  this  time.  Two  other  models  will  be  developed  that 
may  help  to  resolve  these  differences. 


3.  A  MODIFIED  STATE  REPRESENTATION 

The  maintenance  process  and  its  relation  to  the  FFD,  FFI,  and  FFA 
parameters  is  complicated  because  the  state  of  the  system  is  unknown.  In 
fact,  maintenance  actions  are  devised  to  discover  the  state  of  the  sys¬ 
tem.  These  actions  are  imperfect  and  result  in  raisidentification,  non- 
detection.  false  alarms,  and  other  errors.  It  is  this  mismatch  that  we 
are  trying  to  measure.  A  perfect  match  could  be  defined  as  one  in  which 
maintenance  activity  yielded  100  percent  for  FFD  and  FFI  while  yielding  0 
percent  for  FFA.  Despite  the  fact  that  maintenance  is  not  structured  as 
in  a  normal  state  analysis  format,  we  will  proceed  with  a  state  analysis. 

3.1  State  Breakdown  Analysis 


Figure  B-2  shows  the  state  breakdown,  with  each  node  representing  a 
state.  A  logical  place  to  start  is  at  the  system  states  "Fault"  and  "No 
Fault."  Note  that  both  have  a  detection  state  and  a  nondetection  state. 


As  shown  In  Che  next  state,  we  are  interested  in  what  triggered  the  main¬ 
tenance  action.  This  will  become  critical  for  counting  detections.  There 
is  a  critical  point  to  be  made  about  the  detection  phase:  A  nondetection 
of  a  real  fault  will  never  be  noticed  in  the  field.  Awareness  of  the  non¬ 
detection  occurs  only  in  a  perfect  information  state  where  we  Icnow  that  a 
fault  is  inserted.  Actual  faults  will  eventually  be  detected  by  an  opera¬ 
tor,  a  maintenance  history,  or  some  other  means,  or  simply  will  not  matter. 

After  detection,  a  number  of  states  are  available.  Where  there  are 
no  real  faults,  we  are  confronted  with  false  alarms  or  we  trigger  mainte¬ 
nance  in  some  other  system.  The  distinction  is  Important,  because  at  the 
subsystem  level  there  is  not  necessarily  a  false  alarm,  l.e.,  subsystem  A 
manifesting  a  problem  that  is  actually  in  subsystem  B  —  a  real  problem 
exists  and  is  detected  by  A.  The  system  may  contain  both  a  false  alarm 
and  a  nondetection  in  subsystem  B.  This  relationship  highlights  the 
necessity  of  specifying  system  and  subsystem  boundaries  carefully.  If  a 
system  boundary  has  been  drawn  around  subsystem  A,  then  the  detection  just 
described  is  a  false  alarm  since  there  is  no  fault  in  the  system. 

When  faults  are  present  and  detected,  the  state  moves  from  detection 
through  either  system  detection  or  detection  by  other  means.  From  here, 
the  state  may  move  to  either  system  isolation  or  system  nonisolation,  and 
the  FFI  term  is  simply  the  product  of  the  previous  transition  (a)  and 
the  direct  transition  U)  plus  the  non-NSM  detections  (?)  and  the  system- 
isolated  transition  (y) .  The  states  that  exist  for  system  nonisolation 
when  a  fault  is  present  are  the  same  regardless  of  where  detection  occurs. 
Abnormal  isolation  includes  all  correct  isolations  by  other  than  normal 
system  maintenance  and  all  bad  isolations  (including  cannot  duplicate  [CND] 
and  cannot  find  [CNF]).  A  bad  isolation  state  exists  when  the  isolation 
is  to  the  wrong  fault  or  includes  isolation  of  items  without  fault.  In 
this  simplified,  non- real -world,  full-information  model,  false  alarms  are 
given  by  X^  (detection  of  no  fault).  A  recap  is  given  as  follows: 


FFD  =  x 


FFD  =  X  X  a 
s 


FFIs=  Xxax4+Xx?xy 


FFA  =  X, 


FFAS  -  Xj  x  .j 


ICND 
FFA  »  S^r- 


(33) 
(33a) 

(34) 

(35) 

(36) 


where 


3  represents  a  compound  probability  factor  for  a  CND  that  is  the 
result  of  a  nonf allure. 

The  maintenance  action  flow  In  the  real  situation  proceeds  along  dif¬ 
ferent  paths,  because  the  maintenance  personnel  are  unaware  of  the  true 
state.  Figure  B-3  shows  this  flow.  A  maintenance  action  is  Initiated 
when  a  fault  Is  Indicated  or  reported  to  the  maintenance  system.  The  same 
procedure  Is  Involved  regardless  of  whether  detection  was  initiated  by 
normal  system  maintenance,  an  operator  complaint,  or  some  other  means. 

Once  a  fault  is  detected  in  a  subsystem,  a  normal  system  fault-isolation 
procedure  Is  started. 

If  a  fault  Is  correctly  Isolated  to  the  replaceable  unit  containing 
the  fault,  a  repair  action  Is  undertaken.  Following  the  repair,  the  sub¬ 
system  Is  given  a  "go"  check.  If  the  system  "checks  OK."  the  subsystem  is 
returned  to  the  ready  state.  This  sequence  can  occur  from  two  true 
states:  (1)  the  fault  was  correctly  Isolated  to  a  set  of  replaceable 
units  and  repaired,  or  (2)  there  was  an  Incorrect  fault  Isolation  and  an 
unnecessary  repair  was  made.  This  second  action  can  occur  when  there  is 
an  Intermittent  failure  and  an  Incorrect  fault  Isolation  or  when  a  false 
alarm  occurs  and  an  Incorrect  fault  Isolation  leads  to  repair  of  a  replace¬ 
able  unit.  It  should  be  noted  that  a  correct  fault  Isolation  may  result 
In  removals  of  good  replaceable  units  because  of  Inherent  design 
ambiguities. 

Following  a  repair,  the  system  check-out  may  indicate  a  "no  go" 
situation.  This  Indication  can  be  caused  by  several  states,  Including 
(1)  an  Incorrect  fault  Isolation  and  an  unnecessary  repair,  and  (2)  a 
multiple-failure  condition.  The  multiple  failure  may  be  solved  by  multi¬ 
ple  passes  through  fault  Isolation;  at  each  pass,  a  single  fault  Is  Iden¬ 
tified  and  repaired.  This  process  continues  until  the  "go"  check  Is  suc¬ 
cessfully  accomplished. 

One  branch  of  the  flow  chart  shows  the  actions  that  occur  when  normal 
system  fault  isolation  cannot  reduce  the  fault  to  a  set  of  replaceable 
units.  Several  nonstandard  fault-isolation  procedures  may  be  Invoked  — 
for  example,  checking  the  circuits  using  a  voltmeter,  calling  In  contrac¬ 
tor  expertise,  swapping  boxes,  or  simply  doing  "shotgun  repairs."  If  a 
fault  Is  Isolated,  the  replaceable  units  are  sent  to  repair  and  the  system 
Is  checked  out  as  in  a  normal  fault  isolation  and  repair.  Abnormal  fault 
isolation  may  result  in  high  RTOK  rates,  since  good  units  may  be  unneces¬ 
sarily  removed. 

Whenever  nonstandard  fault  isolation  Is  unable  to  locate  failures, 
the  subsystem  Is  generally  submitted  to  a  "go”  check.  If  the  check  Is 
passed,  the  subsystem  is  returned  to  the  ready  state.  This  sequence  can 


Maintenance 

Action 


Perform 

System 

Fault 

Isolation 


be  caused  by  three  events:  (1)  intermittent  failure.  (2)  a  false  alarm, 
or  (3)  an  improper  input  from  a  fault  in  another  subsystem.  In  any  case 
the  subsystem  is  returned  to  the  ready  status.  A  history  check  may  be 
used  to  identify  further  action. 

If  a  "go"  check  has  failed  and  no  fault  has  been  Isolated,  the  sub¬ 
system  is  called  NITS  (not  lsolatable  this  station).  A  NITS  may  occur  for 
subsystems  that  are  designated  for  repair  at  a  different  maintenance 
level.  The  subsystem  remains  in  a  not-ready  status. 

Figure  B-4  summarizes  the  possible  outcome  of  the  maintenance  action 
flow  chart.  One  type  of  maintenance  problem  is  caused  by  intermittent 
failure.  Interraittents  are  generally  discovered  by  keeping  a  history  of 
the  subsystem.  After  several  CNDs.  an  abnormal  (l.e.,  outside  NSM)  fault- 
isolation  action  may  be  initiated  to  discover  the  cause  of  the  problem. 

The  model  that  results  when  actions  and  states  are  combined  is 
depicted  in  Figure  B-5.  The  general  case  is  fairly  complicated  and  helps 
to  illustrate  the  reasons  for  the  traditional  difficulties  encountered  in 
attempting  to  measure  parameters  such  as  FFD,  FFI,  and  FFA.  The  problem 
decomposition  shown  in  Figure  B-5  is  our  model  for  analyzing  the  measur¬ 
able  components  of  FFD,  FFI.  and  FFA.  Again,  faults  that  never  trigger 
maintenance  actions  will  be  ignored. 

The  main  division  occurs  between  maintenance  actions  triggered  by 
normal  system  maintenance  and  those  triggered  by  something  else.  Proper 
fault  detection  occurs  when  normal  system  maintenance  discovers  the 
fault.  Similarly,  proper  fault  isolation  occurs  when  normal  system  main¬ 
tenance  isolates  the  fault  to  the  minimum  number  of  replaceable  units. 

Detection  and  repairs  by  means  other  than  normal  maintenance  will  be 
counted  as  faults  not  detected  or  faults  not  isolated.  For  example,  fault 
isolation  at  the  organizational  level  occurs  when  sufficient  information 
is  known  to  initiate  repair  actions  even  if  good  units  must  be  removed  in 
the  process.  Fault  isolation  ought  to  isolate  to  a  minimum  number  of 
faulty  replaceable  units  in  a  short  time  with  limited  use  of  resources.  A 
proper  fault  isolation  will  be  achieved  when  the  normal  maintenance  pro¬ 
cedures  isolate  to  the  minimum  number  of  replaceable  units  allowed  by  the 
subsystem  design.  An  isolation  will  have  occurred  when  all  faults  have 
been  removed  from  the  system.  The  fault  isolation  illustrated  will  be 
further  broken  down  into  categories  on  the  basis  of  the  type  of  testing 
that  performed  the  isolation:  BIT,  ATE,  manual,  or  semiautomatic. 

False  alarms  are  counted  if  normal  system  maintenance  reports  a  fault 
when  there  is  no  fault  in  the  subsystem.  The  chart  shows  that  CNDs  due  to 
intermlttents  are  clearly  confused  with  false  alarms.  In  addition,  if  the 
fault  was  originally  in  another  subsystem,  it  is  easy  to  imagine  the  main¬ 
tenance  technicians  assuming  a  false  alarm  unless  results  from  the  other 
faulty  subsystem  are  cross-referenced. 
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FIGURE  B-5 .  MODIFIED  STATE  MODEL  FOR  REPAIR  OF  COMPLEX  ELECTRONICS 
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The  final  goal  of  these  analyses  is  to  determine  the  impact  of  main¬ 
tenance  actions  on  readiness.  Figure  B-6  shows  the  readiness  flow  model. 
The  boxed  activity  represents  the  contribution  of  the  maintenance 
activity.  The  main  influence  on  readiness  is  the  time  factor.  For  exam¬ 
ple.  if  a  false  alarm  occurs  but  requires  no  time  or  resources,  it  has  no 
impact  on  readiness.  The  time  factors  are  typically  recorded  as  mainte¬ 
nance  man-hours  per  flight  hour,  mean  time  to  repair,  mean  time  to  fault- 
lsolate,  and  similar  values. 


3.2  Algorithmic  Representations 

The  modified  state  model  provides  a  vehicle  by  which  we  can  begin  to 
develop  the  desired  parameters  algorithmically.  Each  measure  shown  is 
based  on  Figure  B-5  state  and  action  relationships  and  evolves  from  the 
given  definitions. 


3.2.1  Fraction  of  Faults  Detected  (FFD)  -  System  Level 
Theoretical  FFD  is  expressed  as  follows: 


where 


(37) 


s  =  system 

FDg  »  faults  detected  by  the  system 
F  -  all  faults  in  the  system 


and  the  subscripts  T  and  N  indicate  time  and  mode,  respectively,  as  either 
all  events  in  a  time  period  or  all  failure  mode  types  as  matched  with  the 
FMECA.  It  should  be  noted  that  the  latter  may  take  a  significantly  longer 
time  to  obtain  experimentally.  (The  T  and  M  breakdown  is  implicit  in  all 
of  the  following  equations,  and  these  subscripts  are  consequently  deleted.) 

Operational  FFD  is  expressed  as  follows: 

IFDS  =  £MAs  -  IFAS  (38) 

and 

IF  -  IMA  -  I FA  (39) 

where 


MA  ■  maintenance  action 


FA  =  is  false  alarm 

I 
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Consideration  of  subsystems  can  be  accommodated  as  follows: 


(!■) 


where  ssc  refers  to  a  subsystem  contribution. 

The  hierarchy  of  system/subsystera  is  important  here  because  of  the 
proper  and  improper  fault  detections:  It  can  be  seen  that  if  the  subsys¬ 
tem  is  viewed  as  the  system,  then  the  Improper  detections  become  the 
cannot  duplicates  and  the  detections  become  the  maintenance  actions,  so 
that  Equation  40  reduces  to  the  same  set  as  Equations  37  through  39. 

The  system  and  subsystem  detections  can  be  related  by  the  following: 

£FDc  =  l  (£FD  -  £FD  Improper)  (4 


so  that 


/  FD 
I _ ss 

\  l* 


-  l  l  FFD 
ss  \  SSC 


XFDss  improper) 


JFD  improper 
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3.2.2  Fraction  of  Faults  Isolated  (FFI)  -  System  Level 
Theoretical  FFI  is  expressed  as  follows: 


•(!) 


where  FI  represents  faults  isolated. 


(!)  (!) 

(ft)-. 


FFI  =  r  X  FFD 
S  S 


where  r  is  the  system  conversion  of  detections  into  isolations. 


Vi«w 


As  discussed  In  the  set  theory  model,  this  algorithm  tends  to  be  un¬ 
usable  when  we  are  dealing  with  specification,  so  that  a  penalty  of  opera¬ 
tional  FFI  (FFI  )  Is  expressed  as  follows: 

P 

*  *<FIsWt  <46 

-  2(FIS>  -  £RTOKs  -  IVj  S  IIFIS>  -  IBTOKs 

The  purpose  of  the  approximation  is  measurability.  It  overpenalizes 
Improper  isolation  by  a  second-order  RTOK  factor  because  of  a  nondetection 
and  an  Improper  fault  Isolation.  This  cross  product  Is  given  by  Vj  ,  but 

Intuitively  It  is  small  and  also  not  measurable  under  current  reporting 
systems:  It  Is  therefore  assumed  to  be  0  In  the  approximation. 

Consideration  of  subsystems  can  be  accommodated  as  follows: 


(*) 


£FIs  =*  !FDss  proper  +  X<FI_.  *  ss) 
XFIS  =  XFDss  proper  +  cXFDss  Improper 


where  c  is  the  fraction  of  improper  detections  that  are  converted  to 
system  Isolations.  This  can  be  expanded  as  follows: 

£FI  =  X (FD  -  FD  Improper)  +  cXFD  improper 
s  ss  ss  ss 

XFI  *  X(FD__  +  (1  -  c)  FD  Improper) 
s  ss  ss 

3.2.3  Fraction  of  False  Alarms  (FFA)  -  System  Level 


Theoretical  FFA  Is  expressed  as  follows: 


\XFA  *  xf; 


The  denominator  Is  modified  to  control  the  range  of  the  FFA  parameter  to 
be  between  0  and  1. 

Operational  FFA  Is  expressed  as  follows: 


(by  substitution  of  Equation  39) 


s 


[•Oil! 


s  proper 


&slcms 


IFA  «  0XCND 


(51a) 


The  CNDs  are  the  "cannot  duplicate"  maintenance  events;  £  is  an 
empirically  derived  coefficient  that  relates  system-generated  CND  values 
to  false  alarms  (see  Equation  36a) . 

Consideration  of  subsystems  can  be  accommodated  as  follows: 


SS  \  ^ss  +  ZFSS 


I  (fa  )  -  X  ?  T1;)  (fass\. 
1-1  \  ss/l  i-1  j=l  \  /• 


m 

uys&r, 

OH 


*.  y  *. 


iM 


where  is  the  cross-detection  of  subsystem  1  as  a  result  of  failures 
in  subsystem  J  (note:  -  0). 

3.2.4  Algorithmic  Summary 

The  final  operational  definitions  are  given  by: 

/  IMA  -  £_ICND  \ 

FFD  =  (  — r5 - - —  < 

\  IMA  -  PlCND  / 


XMA  -  0XCND 


/  XFy  IRTQKs  \ 

\IMA  -  0XCND  / 
/p  XCNDS\ 

Vf FFA 


(combination  of 

Equations  46.  39,  and  51) 


/ bxcnd\ 

YMA 


s»: 


FFA„  * 
s 


(combination  of 
Equations  40  and  41) 


(57) 


Finally, 


FAR  -  ^  |  ICND  (58) 


where  T  is  a  time  measure. 

3.3  Measurability  and  the  State  Representation 

Equations  54  through  58  are  derived  in  terms  of  measurable  quantities 
from  maintenance  reporting,  with  the  exception  of  0  and  RTOK.  RTOK  can 
be  measured  elsewhere,  but  0  is  not  available  by  any  means  currently  and 
may  be  a  complex  function  of  many  factors  (including  desire  on  the  part  of 
the  maintenance  crew) .  The  approach  to  be  taken  with  this  model  is  to 
choose  a  particularly  robust  data  set.  The  coefficients  0  and  0g  can  be 

computed  by  regression  for  the  robust  data  set,  and  their  relationships  can 
be  determined.  While  this  is  easy  to  assert,  it  may  not  be  so  easy  to  do. 

Of  particular  concern  is  the  interaction  and  interdependence  of  the 
three  terms.  Although  0  was  derived  as  a  coefficient  in  the  determina¬ 
tion  of  false  alarms  (Equation  51),  the  interaction  occurs  in  all  three  of 
the  principal  terms  (Equations  54  to  58).  It  would  make  the  entire  model 
critically  linked  to  the  proper  determination  of  0  and,  hence,  to  false 
alarms.  It  has  been  noted  that  the  false  alarm  is  the  most  difficult  to 
measure. 


4.  A  FLOW  MODEL  REPRESENTATION 

The  preceding  state  representation  has  been  useful  in  showing  the 
essence  of  the  maintenance  process,  but  still  has  some  shortcomings  in 
representing  false  alarms  and  time-dependent  parameters.  The  state  model 
has  only  an  indirect  interface  to  readiness.  The  flow  model  was  developed 
to  help  provide  insight  into  these  processes.  The  flow  model  is  an 
attempt,  at  least  in  a  generic  sense,  at  capturing  the  flow  of  events  in  a 
maintenance  system.  Further,  the  development  of  the  reporting  process  can 
be  modeled  into  this  representation.  Figure  B-7  shows  the  gross-level 
breakdown  of  the  flow  model.  It  consists  of  six  modules  with  appropriate 
submodules: 

0.  Readiness/Avai lability 

0a.  Mission  Availability  Interface 

1.  Mission  and  Through-Mission  Activity 

2.  Mission  Activity 

3.  Post -Miss ion  Activity 

4.  Unscheduled  Maintenance  Activity 
4a.  Resource  Allocation  Activity 
4b.  Troubleshooting 

4c.  Logistics  Interface 
4d.  Awaiting  Parts 

5.  Scheduled  Maintenance  Activity 
5a.  Resource  Allocation  Activity 
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Each  module  Is  represented  by  a  flow  process  and  will  be  discussed  In 
detail  In  the  following  subsections. 

4.1  Some  Mathematical  Basics 

The  interface  between  Readiness/Aval lability  and  Hlsslon/Aval lability 
Is  shown  In  Figure  B-8,  which  contains  most  of  the  modules  and  types  of 
flow  mechanism  shown  In  the  model.  Since  this  model  Is  flow-driven,  we 
need  to  define  a  measurement  base.  Also  note  that  any  flows  will  be 
controlled  by  the  decision  gates  (diamonds  In  the  figures). 

Let  *  represent  a  flow  parameter  for  any  element  1. 

Let  *  represent  the  yes  output  and  u  represent  the  no  output  of  a 
decision  box. 

then 

* 

*3a  6  represents  the  yes  output  rate  of  decision  box  3a, 6. 

$2a  4  represents  the  flow  to  box  2a. 4  (no  superscript). 

The  model  will  be  written  In  terms  of  the  previously  defined  defini¬ 
tions  and  on  a  per-system  basis.  That  is,  each  term  refers  to  the  rates 
of  accumulation  by  only  one  system  at  a  time.  In  a  hierarchical  sense, 
the  whole  aircraft  could  be  such  a  system;  but  one  LRU  or  SRU  could  also 
be  a  system,  depending  on  the  Introspection  level  required.  Some  rela¬ 
tionships  follow. 

4.1.1  Decision  Mode 

For  a  decision  node  (assumed  nonaccumulating); 


This  assumes  that  decision  nodes  do  not  accumulate  anything  but  only  pass 
through  In  a  branching  form. 

4.1.2  Other  Modes 

For  other  nodes: 


(example  0a,5  in  Figure  B-8) 


For  accumulating  nodes: 


♦out  *  JC<*ln'n)dt  {60) 

where  C  Is  the  rate  at  which  the  accumulation  is  processed. 

Where 

161  > 


4. 1.2. 2  Nonaccumulatlng  Nodes 
For  nonaccumlatlng  nodes: 

♦in  ’  ♦out 
which  implies 
n  =*  0 
and 


(62) 


(63) 


C(^in) 


(64) 


4.1.3  Absorbing  Nodes 
For  absorbing  nodes: 


♦ 


out 


0 


(example  2,10  in  Figure  B-ll) 

(65) 


This  implies 

C(*,n)  -  0  (66) 

and 

n  *  (67) 
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This  Is  sometime  called  a  sink. 


4.1.4  Creating  Nodes 
For  creating  nodes: 


4> 


In 


0 


(example  0.0  In  Figure  B-8) 


(68) 


This  implies 


C(O.n)  -  *QUt 


(69) 

(70) 


This  Is  sometimes  called  a  source. 

The  exact  function  of  C  may  have  several  forms,  and  empirical  data  may  be 

needed  to  best  describe  it. 

4.2  Time  Representations,  Statlonarltv,  and  Steady  State 

In  the  accumulation  of  parameters  throughout  the  system,  the  time 
lags  are  significant  to  the  transient  behavior,  and  the  instantaneous 
response  may  vary  widely  over  time.  When  one  considers  the  Input  rate  to 
a  series  of  nodes  at  time  t.  the  output  response  occurs  at  time  (t  *■  r). 
where  t  represents  the  amount  of  time  It  takes  the  system  to  respond  to 
this  input.  In  order  to  make  an  expected  value  statement,  one  must  accu¬ 
mulate  data  over  a  period  of  time  sufficient  to  include  normal  cycles 
present  in  the  system.  For  example,  the  number  of  inputs  over  this  time 
could  be  represented  as: 

"l  •  fo  *in  dt  17 

and  the  number  of  outputs  by 

t+T 

“o-J,  «outdt  11 

In  order  to  keep  the  integral  limits  the  same  when  comparing 

with  N  ,  we  must  assure  that  the  length  of  time  is  long  enough  that  the 
o 

transient  effect  of  t  is  small  to  the  problem.  That  is,  t  +  r  -*  t. 
and  the  data  periods  are  statistically  significant.  This  can  be 


Illustrated  by  Figure  B-9,  which  shows  the  time  Integrals  as  areas.  For 
the  data  shown.  4  is  assumed  always  positive  and  the  integrals  are 
related  by  the  shading.  The  assumption  is  that  the  integral  over  0  to  t 
and  the  Integral  over  t  to  t  +  t  are  approximately  equal: 

t  t+T 

I  edt  a  /  $dt  (1 

0  T 


t+T  O  t  t+T 

f  <$dt  =  /  (fdt  +  f  ;}>dt  +  /  $dt 
T  T  o  t 

t 

3  /  <}>dt 


/Oj=>  /  d>dt  +  /  <J>dt  3 


FIGURE  B-9.  INTEGRAL  REPRESENTATIONS 


There  are  three  conditions  that  must  be  satisfied  for  this  to  be  true: 


1.  *  must  be  positive.  This  is  true  for  the  maintenance  problem, 
where  negative  flows  to  the  nodes  have  no  physical  meanings. 

2.  The  integral  must  be  significantly  greater  than  zero.  That  is, 
the  integrand  cannot  be  small  over  a  large  portion  of  the  data. 
This  prevents  the  small  shaded  regions  of  the  integral  from 
approaching  a  significant  part  of  the  total  integral. 

3.  t  »  r.  For  the  maintenance  problem,  t  is  on  the  order  of 
MTTR,  so  that  t  must  be  many  multiples  of  MTTR. 

Finally,  representativeness  places  one  further  requirement  on  the 
data.  If  there  are  natural  cycles  (such  as  seasonal  changes),  the  data 
should  be  taken  over  complete  cycles 'or  a  large  number  of  cycles  so  that 
the  effect  of  partial  cycles  does  not  significantly  affect  the  answer. 
With  these  requirements  enforced  on  data  sets,  we  will  drop  the  integra¬ 
tion  limits  from  notations  in  equation  development.  Further,  any  rate 
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For  the  flow  model,  the  conservation  of  flow  through  the  system  and 
the  total  flow  through  a  node  is  the  governing  law.  For  example,  the  flow 
Into  a  node  Is  equal  to  the  sum  of  all  Its  parts  with  some  exceptions: 


I  <*ln 

ss  me 


-  <p  ) 

error 


where  4  error  represents  a  misplaced  repair  action.  This  misplaced 
repair  action  causes  a  counting  of  a  single  event  to  occur  in  two  of  the 
subsystem  models  (e.g..  an  Improper  isolation  of  subsystem  A  [fault  In 
subsystem  B]  is  a  false  alarm  when  we  are  dealing  only  with  A  as  the 
"system-) . 

4.4  Readiness/Aval lability  and  Mlsslon/Avallabllltv  Interface 


Figure  B-8  shows  the  module  to  be  discussed. 

This  Interface  module  starts  with  the  system  operational  available 
pool  (0  ,6  of  the  figure).  This  pool,  at  any  given  time,  may  or  may  not 

d 

contain  systems  for  mission  tasking.  On  the  left  Is  the  periodic  check  of 
systems  for  scheduled  maintenance,  such  as  calendar  checks.  Note  that 
this  check  is  also  made  as  systems  return  to  the  availability  pool  from 
repair  or  mission  completion.  Also  in  this  loop  is  the  analysis  decision 
for  maintenance,  and  this  would  Include  repeated  gripes  of  CND  or  non¬ 
mission-critical  needed  repairs,  or  Just  a  trend  noted  by  the  maintenance 
chief.  This  Is  outside  normal  system  maintenance  (NSM).  The  loop  on  the 
right  Is  for  mission  tasking.  A  mission  Is  called  on  the  basis  of  mission 
requirements  due  to  either  Internal  or  external  tasking.  For  purposes  of 
statlonarlty  we  will  assume  that  external  tasking,  at  least  In  the  near 
term,  is  not  affected  by  the  organizational  maintenance  flow.  Internal 
tasking  may  be  effected  by  any  of  the  mission  elements  or  by  a  “no"  answer 
to  system  availability. 

For  the  readlness/avai lability  interface,  the  operational  availabil¬ 
ity  (Aq)  can  be  given  by 


(no  dimensions) 


[since  Oa.4  is  a  decision  node 
(using  Equation  59)] 


(76 


Simply  stated,  this  gives  the  operational  availability  as  the  per¬ 
centage  of  requests  for  missions  that  can  be  filled  over  a  time  period. 

♦ 

One  other  term  from  this  chart  that  will  be  needed  later  is  *oa,8  which 
is  the  readiness  detection  of  failures,  which  is  outside  NSM.  This  is 
given  by 

♦ 

RF _ *  a  (rate  terra) 


The  Redball  activity  in  Oa.IO  is  discussed  in  the  next  section. 

4.5  Pre-Mission  and  Throuqh-Mission  Activitv 

The  detailed  breakdown  of  the  pre-mission  and  through-mission 
activity  is  given  in  Figure  B-10.  The  pre-mission  activity  will  include 

maintenance  and  operator  pre-mission  checks  and  may  Include  Redball1 
activity,  which  is  a  last-ditch  attempt  to  get  a  mission  off.  This  Red- 
ball  activity  may  Include  cannibalization  if  necessary,  and  it  represents 
a  mini-form  of  the  unscheduled  maintenance  activity.  This  is  done  with 
little  or  no  reporting  (at  least  directly).  Parts  cannibalization  during 
the  Redball  activity  will  be  charged  to  the  cannibalized  system  and  appear 
as  a  multiple  gripe.  While  this  activity  appears  to  hopelessly  complicate 
the  maintenance  flow,  it  is  somewhat  self-compensating  in  that  the  fail¬ 
ures  and  repairs  are  eventually  reported,  albeit  against  another  system. 
The  Redball  activity  also  appears  in  the  periodic-check  loop  of  the 
read! ness/ aval lability  interface  because  the  cannibalization  might  be  from 
an  available  system,  thus  removing  it  from  the  available  systems  pool. 

Two  Important  variables  noted  in  the  pre-mission  phase  are  the  prob¬ 
lem  detection  by  non-NSM  at  1,5: 


(rate  terra) 


the  pre-mission  reliability  (R  ) : 

put 

,  IVlO  ,  ;*  1.10 

m  '*1  1  , 

Oa.4 


1TAC  term:  MAC  and  SAC  use  Red  Streak.  SAC  had  previously  used  a  Blue 
Streak  designation. 


By  use  of  Equation  75 


/♦ 


1.10 


pn  Jo  A 
0a,4  o 


and 


JO 


A.  X  R 


1.10 


o  "pm  J4» 


0a,  4 


4.6  Mission  Activity 

The  detailed  breakdown  of  mission  activity  Is  shown  In  Figure  B-ll 
The  MAC  loop  covers  segmented  missions  and  repairs  during  the  mission, 
such  as  MAC  activities.  This  section  Is  the  most  robust  In  terms  of 
descriptors  of  system  readiness.  The  overall  system  reliability  (R)  Is 
measured  as 


JO 


R  = 


2.5 


Jow 

2,5  =  _ 

Jo  JO* 

1.1  0a.4 


Survivability  (S)  Is  measured  as 


JO 


S 


2.4 


Jo 


2.4 


JO 


2.3 


Jo 


2.4 


JO 


2.5 


Jow 

2.5 


and  conditional  effectiveness  (E)  is  measured  as 
1  _  J*2,3  . 


JO 


Oa.4 


conditioned  upon  having  capability. 


These  several  parameters  can  be  related  as 


The  simplified  model  to  this  point  has  assumed  that  the  mission  Is  a 
go/no-go  decision.  In  dealing  with  subsystems,  this  is  normally  ade¬ 
quate.  However,  In  dealing  with  full  systems  (such  as  aircraft),  mission 
capability  Is  usually  broken  down  by  full,  partial,  marginal,  and  not- 
mission-capable,  so  that  the  conditional  effectiveness  will  be 

«-  *ol  >VS1  « 


or  the  component  products  are  summed  over  the  1  states  involved.  This  is 
In  direct  agreement  with  the  WSEIAC  model1. 

4.7  Post -Mission  Activity 

The  post-mission  activity,  detailed  In  Figure  B-12.  contains  several 
parameters  of  Interest. 

The  post-mission  failure  reporting  outside  MSM  is  given  by 

"W  "  *3.3  (8i 

From  here,  we  can  write  the  total  of  all  failure  reporting  rates 

outside  normal  maintenance  as  maintenance  actions  not  NSM  *  MA  ' : 

NNS 

MJl  -  MP  +  PM  +  RF._  (combined  Equations  (8« 

NNS  NS  NS  NS  77,  78,  and  88) 


•  +  ♦  ♦ 
"NnS  ■  *3,3  +  *1.5  +  *0a,8 


The  mission  returns  to  the  available  pool  (L.)  is 


*W  ■  *3.4 


1 Weapon  System  Effectiveness  Industry  Advisory  Committee,  Chairman's 
Final  Report.  AFSC-TR-65-6 ,  January  1965. 
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and  the  total  up  mission  returns  (U^)  Is 


$3(4  +  ^3,4 


A,_  +  A__ 
MR  TM 


where  A  Is  the  through-mission  availability,  which  may  need  adjustment 

Tn 

for  Redball  action. 

This  concludes  the  analysis  of  the  readiness/availability  side  of  the 
flow  model. 

4.8  Maintenance  Activity 

The  scheduled  and  unscheduled  maintenance  activities,  together  with 
the  resource  allocations,  are  shown  in  Figure  B-13. 

4.8.1  Unscheduled  Maintenance  Activity /Scheduled  Maintenance  Activit 


and  Resource  Allocation 


The  resource  allocation  actually  resides  in  both  the  scheduled  and 
unscheduled  maintenance  activity,  and  the  two  will  be  handled  together. 
Figure  B-13  shows  the  flow  model  of  these  activities.  Important  rates  to 
be  examined  occur  at  the  start  of  unscheduled  and  scheduled  maintenance 

activities  (MA  and  MASChed) •  We  have  not  subscripted  the  unscheduled 

maintenance  activity,  because  this  is  the  variable  that  will  be  examined 
most  closely.  These  terms  are  given  by 


^sched  *5.1 


MA  =  <|>. 


ML  »  ,  -  MA  +  MA 

total  sched 


Among  other  interesting  parameters  are  the  logistics  interface  param¬ 
eters  and  resource  allocation  parameters,  but  these  will  not  be  dealt  with 
in  detail  here. 


The  unscheduled  maintenance  activity  has  two  components.  Those  which 
occur  from  normal  system  maintenance  are  called  MA^g^.  Those  which  occur 

from  outside  normal  system  maintenance  are  called  MA^^: 


*  a  +  “Sjsm  <96) 

! 

I 

|  MA  *  MP^  +  PM^  +  RF^  +  MRjjgjj  (from  Equation  89)  (97) 


4.8.2  Troubleshooting  Activity 

The  troubleshooting  activity  is  given  in  Figure  B-14.  From  the 
troubleshooting  activity,  we  see  that  CND  Is  given  by 

*  *4b,9  +  ^4b,6  +  ^4b,8  +  *4b,5  +  *4b,13  (98) 


Of  these,  a  portion  of  the  CNDs  and  some  portion  of  the  NRTS  are  taken  to 
be  real  failures.  The  false  alarm  rate  will  then  be 


FA  -  [p  CND  +  p  j  NRTS] 

(99) 

FAR  -  i  /FA 

(100) 

where  B  Is  the  undefined  (but  assumed  stationary)  coefficient  representing 
the  portion  of  CNDs  that  are  not  real  failures,  and  fl^  Is  the  undefined 

(but  assumed  stationary)  coefficient  representing  the  portion  of  NRTS  that 
are  not  real  failures.  T  Is  the  Integration  period  or  mission-hour  figure. 
With  warranty  Ignored  for  the  moment,  B^  approaches  zero  because  "bad 

actors"  are  nearly  always  real  faults.  Some  exceptions  to  this  might 
Include  improper  test  procedures  or  test  tolerances.  NRTS  from  nonisolat- 
able  but  verified  faults  have  a  high  probability  of  being  real  faults. 

One  other  component  Is  the  isolatable  but  not  repairable  NRTS  shown  in 
Figure  8-15.  Warranty/guarantee  (W/G)  on  the  other  hand,  will  be  a  false 
alarm  to  the  extent  of  good  returns  from  warranty  service  (very  hard  to 
measure) : 


FAW/G  "  “"Sl/G 


(101) 


so  that  for  *  0  in  all  but  warranty/guarantee,  the  false  alarm  is  given 
by 

FA  =  [fJ(CND)  +  RTOK^J  (102) 


If  one  assumes  that  the  warranty/guarantee  RTOK  rate  approaches  zero  (this 
is,  after  all,  one  purpose  of  a  warranty)  or  is  unmeasurable,  then 


FA  *  p  CND 

The  fraction  of  false  alarms  will  be  given  by 


(103) 


JFA  pjCND 

FFA  *  -  =  - 

/MA  /HA 


(104) 


This  excludes  the  scheduled  maintenance  actions. 

Figure  B-14  can  be  used  to  generate  the  other  primary  parameters. 
The  fraction  of  faults  detected  will  be  given  by 


FFD  * 


(105) 


where  F  is  the  rate  at  which  faults  occur  and  FD„  is  the  rate  at  which 

s 

faults  are  detected.  If  the  rate  functions  are  based  on  operating  hours, 
then 


F 


1 

MTBF 


4>,  .  -  FA  +  U 
4 , 1 


and 


(106) 


F  -  MA  -  FA  +  0  (107) 

where  U  is  the  rate  of  failure  occurrences  that  do  not  trigger  maintenance 
actions  (a0:  see  set  theory  discussion  on  relevance).  This  reduces  to 


F 


MA  -  FA 


(108) 


F  »  MA  -  P  CND 


(using  Equation  103) 


(109 


The  detections  can  be  related  through  the  rate  of  maintenance  actions 
triggered  by  the  system  and  the  false-alarm  rate  as 


FD_  ■  MA  -  FA., 
s  s  s 


FDS  »  MAs  -  f>sCNDs 


(111) 


J(MAS  -  gsCNDs) 
J(MA  -  0  CND) 


(112) 


The  distinction  between  B  CND  and  just  BCND  is  that  the  subscript 

s  s 

represents  NSM-generated  maintenance  actions. 

Isolation  may  be  similarly  written  by 


J(MA  -  FA) 


where 


FIs  *  *40.5 


•  •  • 


MA-FA-HA-pCND 


f(MA  -  0  CND) 

This  equation  suffers  from  the  inherent  specification  problems  pointed  out 
in  each  of  the  previous  models,  so  that 


£ (FIs^ proper 
J(MA  -  3  CND) 


r  r  r  +  f/Afjr 


s  proper 


f(FI_  -  RTOK  ) 


r  J (MR  -  P  CND) 


We  can  also  note: 


r  (fi  ) 

1  s  proper 


FFI  *  r  x  FFD 
p  s 


'<FVproper 


where  r  -  conversion  of  detection  rate  to  isolation  rate, 
s 

Since  the  bulk  of  the  mathematical  parameters  of  Interest  have  been 
developed  to  this  point,  the  remainder  of  the  model  will  be  discussed  only 
in  passing  for  completeness. 

4.8.3  Logistics  Interface 

Figure  B-15  shows  the  logistics  Interface  model  and  includes  incoming 
and  outgoing  systems  and  parts,  as  well  as  inspection  and  tests  and  the 
repair  actions.  Two  measures  of  note  are  generated  by  this  section.  The 
mean  time  to  repair  can  be  computed  by  tracking  repairs  through  the  sys¬ 
tem.  It  should  include  only  those  which  are  repaired.  The  MTTR  can  be 
given  by 


where  Tj  Is  the  exit  time  measured  at  node  4c. 4  and  t^  is  the  entrance 

time  at  node  4.1.  In  general,  there  does  not  exist  a  t^  for  every  T^. 

because  there  are  exit  paths  other  than  repair,  so  that  i  should  include 
only  completed  repairs.  The  second  term  of  Interest  is  the  .RTOK  rate, 
measured  in  the  returns  from  intermediate  maintenance: 

RTOK  =  u  (124 


This  is  the  basis  of  the  term  that  appears  in  Equation  118,  where 


RTOKs  =  5RTOK 


(125 


If  the  RTOK  rate  is  assumed  to  be  in  proportion  to  the  detection 
rate,  then 

5  a  FFD  (126 


and 


RTOK  s  FFD  x  RTOK 
s 


(127 


The  approximation  is  made  to  avoid  a  difficult  measurement.  The 
rationale  is  as  follows: 

1.  Returns  from  intermediate  maintenance  that  are  tagged  no  fault 
found  (NFF)  are  RTOKs.  These  are  sent  from  organizational-level 
maintenance  to  intermediate-level  maintenance  as  a  result  of  the 
following  sequence  of  events:  a  detection  followed  by  a  verifi¬ 
cation  followed  by  an  isolation. 

2.  This  process  leads  to  the  conclusion  that  RTOK  is  associated  with 
verified  faults  to  a  high  level  of  confidence. 

3.  The  FFD  term  is  a  natural  partition  between  system  fault  indica¬ 
tions  and  fault  indications  outside  NSM. 


4.  The  proportionality  of  faults  to  RTOK  partitioning  can  therefore 
be  assumed  to  be  the  same  proportionality  as  detections  to  faults 


4.8.4  Awaiting  Parts  and  Cannibalization 


Figure  B-16  shows  the  Inventory,  parts  supply,  and  cannibalization 
representation.  The  primary  parameters  of  Interest  here  relate  to  the 
cannibalization  and  Inventory  questions.  For  example,  the  ratio  of  repair 

conversions  directly  from  inventory  to  cannibalization  and  back  order  (R) 
is  given  by 


_  J^d.2 

**4d,5 


(128) 


This  term  must  be  controlled  through  inventory.  If  R  =  0,  then 
spares  are  probably  too  numerous  or  logistics  pipelines  are  very  small. 
It  would  appear  that  R  =  e+.  where  c  is  sized  in  relation  to  mission 
criticality,  failure  rate,  and  other  repair  systems,  would  be  a  design 
goal.  Finally.  R  »  c+  indicates  a  spares  or  logistics  problem. 

4.9  Flow  Model  Algorithmic  Summary 

The  parameters  of  interest  (FFD,  FFI,  and  FFA)  are  summarized  below 
for  the  flow  model. 

Fraction  of  Faults  Detected  (FFD) 


FFD 


f(MAs  -  PSCNDS) 
J(MA  -  0  CND) 


Fraction  of  Faults  Isolated  (FFI) 


(112) 


FFI 


J (MA  -  8  CND) 


and 


(116) 


FFIp 


J (FI  -  FFD  x  RTOK) 
4  s _ 

J (MA  -  8  CND) 


(combined  Equations  119  and  127)  (129) 


I 


Finally, 


gjCND 

JMA 


FAR  3  §  JCND 


(combined  Equations  100  and  103) 


4.10  Measurability  and  the  Flow  Reoresentatlon 


The  total  Integrated  flow  representation  with  cross-references  to  the 
Individual  figures  is  provided  in  Figure  B-17.  The  preceding  equation  set 
can  be  used  to  measure  the  desired  parameters  in  terms  of  maintenance 
reporting.  Individual  reports  will  Include  CND,  MA,  and  FI  events  from 
which  numerical  rates  can  be  developed.  Summary  data  collection  systems 
often  report  these  rate  terms  for  certain  periods  of  time.  However,  two 
measures  are  not  easily  developed:  the  coefficient  fl,  which  represents 
the  partition  between  false-alarm  CND  and  real-fault  CND;  and  the  RTOK 
rate,  which  may  be  lagged  by  a  significant  amount  of  time  from  the  other 
maintenance  activity.  B  may  be  discernible  In  a  particularly  robust  data 
set.  We  have  three  basic  methods  of  accounting  for  RTOK  rate: 

1.  Measure  over  extremely  long  periods  of  time. 

2.  Assume  statlonarlty  (this  assumption  needs  checking). 

3.  Compute  the  time  shift  and  utilize  different  Integration  limits 
on  the  RTOK  data. 


5.  MODEL  INTERACTIONS  AND  CONSEQUENCES 

Table  B-l  summarizes  the  key  parameters  as  developed  in  each  of  the 
three  models.  All  three  models  agree  on  functionality.  For  example.  FFA 
Is  a  function  of  CND  and  maintenance  actions  In  each  of  the  models.  The 
form  of  each  key  parameter  is  identical.  Each  model  points  out  that  It  Is 
important  to  know  what  triggers  maintenance  activity  and  how  fault  Isola¬ 
tion  was  achieved.  These  latter  two  items  are  not  consistently  recorded 
at  present  in  the  AFTO  reporting  system.  Each  model  further  points  out 
the  difficulty  in  relating  CND  to  false  alarms.  In  the  set  theory  model, 
it  is  manifested  by  CNDj  for  improper  CND.  In  each  of  the  state  and  flow 

models,  it  Is  manifested  by  an  empirical  coefficient  (B),  which  represents 
a  partitioning  of  the  CNDs.  Each  model  has  also  pointed  out  the  criti¬ 
cality  of  the  false-alarm  measure.  In  every  case,  each  of  the  parameters 
of  Interest  requires  an  accurate  measure  of  false  alarm  to  be  considered 
accurate.  Finally,  each  of  the  models  has  pointed  out  separate  insights 
Into  the  process.  The  set  theory  model  forces  an  explicit  statement  of 
assumptions  that  are  Inherent  in  all  three  models  but  not  explicitly 
stated  (such  as  2RTOKu,  that  is.  RTOK  due  to  unnecessary  repairs  are 

assumed  to  be  zero).  The  set  theory  model  further  provides  a  method 
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of  specifying  what  should  be  measured.  The  flow  model  representation  pro¬ 
vides  a  direct  link  between  the  maintenance  model  and  readiness  and  has 
pointed  out  the  limits  that  we  must  place  on  data  gathering  In  terms  of 
time  sufficiency,  cycle  breakdown,  and  quantity  of  data.  Finally,  because 
of  Its  Inherent  simplicity  and  conformity  with  current  data  gathering,  the 
modified  state  model  represents  the  best  computational  fit.  This  fit 
notwithstanding,  there  are  several  areas  of  concern: 

-  The  relationship  between  RTOK  and  RTOKg  is  critical.  We  have  an 
estimator  derived  in  the  flow  model  (i.e..  RTOK  *  FFD  x  RTOK). 

5 

This  estimator  will  allow  us  to  proceed  on  measured  RTOK  data  or 
RTOK  rate  data.  Further,  the  flow  model  has  shown  that  If  data 
time  and  sufficiency  requirements  are  met,  we  can  make  this 
measurement  without  Invoking  serial  number  tracking. 

-  We  do  not  have  an  estimating  procedure  for  the  relationship  (fl  ) 

5 

between  system-generated  CND  and  system-generated  false  alarms. 

3s  may  be  Inherently  related  to  Mbad  actors"  that  are  identified 

by  system  fault  Indications.  This  would  require  serial  number 
tracking. 

-  We  do  not  have  an  estimating  procedure  for  the  relationship  (3) 
between  total  CND  and  false  alarms.  An  estimate  could  be  made  by 
relating  past  CNDs  to  "bad  actor"  determinations.  This  would 
require  serial  number  tracking. 

All  three  models  point  out  what  we  cannot  do  with  field-measured 
data,  unless  we  are  willing  to  devote  Inordinate  amounts  of  resources. 

That  Is.  an  analyst  would  be  required  on  site  at  the  organizational  level 
to  define  failures  that  are  undetected  by  any  means,  or  false  alarms  that 
are  recognizable  as  such  and  Ignored  and  do  not  trigger  a  maintenance 
action.  These  two  events  have  been  placed  In  the  nonrelevant  category  not 
only  because  they  can  Justifiably  be  considered  not  relevant,  but  also 
because  they  cannot  be  measured. 

A  final  note  has  to  do  with  the  breakdown  in  fraction  of  faults  iso¬ 
lated  that  has  provided  us  with  two  measures:  FFI  and  FFIp.  While  the 

former  may  be  more  easily  related  to  design  variables.  Its  value  for 
specification  is  questionable,  because  it  can  be  easily  manipulated.  The 
latter,  while  attractive  for  specification  purposes,  is  much  harder  to 
measure  and  may  not  be  directly  relatable  to  design  measures. 

The  value  of  3  Is  currently  not  measurable  and  enormously  compli¬ 
cates  all  three  of  the  measures.  An  arbitrary  assignment  of  3  *  1.0  may 
be  used  for  specification,  since  it  really  only  separates  CNDs  that  are 
due  to  inability  to  verify  faults  from  CNDs  that  are  due  to  bad  detec¬ 
tion.  Both  of  these  are  negative  properties.  Care  must  be  taken,  how¬ 
ever,  to  report  CNDs  accurately;  and  If  the  latter  course  is  taken,  it 
must  be  recognized  that  FAR  no  longer  fits  intuitive  or  widely  recognized 
definitions.  Finally,  an  arbitrary  3  »  1.0  may  reduce  the  ability  to  pre¬ 
dict  any  of  the  measures. 


RADC-TR-83-4,  January  1983.  UNCLASSIFIED 


The  objective  of  this  study  was  to  develop  an  analytical  base  of 
methodology  and  procedures  to  be  used  In  the  testability  area.  Testability 
Is  a  subset  of  system  maintainability  and  is  defined  by  the  system  fault- 
detection  and  isolation  capability.  During  this  effort  a  comprehensive 
survey  of  the  open  and  closed  literature  was  performed  to  identify  the 
analytical  concepts,  models,  algorithms,  and  definitions  currently  used  in 
the  testability  area.  The  report  includes  a  comprehensive  listing,  defi¬ 
nition.  and  discussion  of  commonly  used  testability  parameters  and  their 
components . 


ARINC  Research  Corporation.  Maintainability  Prediction  and  Demonstration 
Techniques.  573-01-2-1032,  March  1970.  UNCLASSIFIED 

A  study  was  performed  to  develop  and  validate  improved  maintainability- 
prediction  and  -demonstration  techniques  for  use  on  all  major  classes  of 
Air  Force  electronic  systems  at  the  on-equipment  level  of  maintenance.  The 
prediction  techniques  are  design-oriented  and  predict  individual  categories 
of  time  (preparation,  fault  location,  item  obtainment,  fault  correction, 
and  preventive  maintenance). 


ARINC  Research  Corporation.  Special  Report  on  Operational  Suitability 
(OS)  Verification  Study  Focus  on  Maintainability.  1751-01-2-2395,  February 
1981 .  UNCLASSIFIED 

This  report  presents  findings  of  an  investigation  of  system  maintain¬ 
ability  assessment  for  USAF  tactical  fighter  aircraft.  A  typical  mission 
turnaround  cycle  (MTC)  is  presented,  and  the  driving  elements  of  MTC  time 
and  maintenance  resources  are  identified.  Special  emphasis  is  placed  on 
the  assessment  of  troubleshooting  characteristics;  specifically,  testabil¬ 
ity  parameters  of  time,  accuracy,  and  thoroughness  are  identified.  A  con¬ 
cept  of  testability  design  assessment  is  outlined. 


Baran.  Harry  A.  (School  of  Systems  and  Logistics,  Wright-Patterson  Air 
Force  Base) .  Effect  of  Test  Result  Uncertainty  on  the  Performance  of  a 
Context-Free  Troubleshooting  Task,  LSSR  86-82,  17  December  1982. 
UNCLASSIFIED 

This  paper  attempts  to  determine  whether  human  bias  exists  under  con¬ 
ditions  of  test  result  uncertainty  such  that  troubleshooting  performance 
is  systematically  affected.  A  knowledge  of  such  bias  might  be  useful  in 
assessing  the  utility  of  powerful  signal  detection-in-noise  analytical 
tools,  e.g..  the  ROC  curve  analysis,  to  improve  predictions  of  trouble¬ 
shooting  performance  and  reduce  troubleshooting  error  by  allowing  man-made 
machine  troubleshooting  systems  to  be  optimized  on  the  basis  of  the 


response  characteristics  of  both  machine  and  man.  This  paper  contains 
considerable  discussion  of  false  alarms  and  percentages  of  all  types  of 
Isolation  errors. 


Boring.  G.,  and  Rayburn.  J.  (ARINC  Research  Corporation).  AN/APS-96  Radar 
System  Product -Improvement  Program.  1173-01-1-1677,  November  1977. 
UNCLASSIFIED 

This  report  describes  significant  engineering  improvements  in  the 
AN/APS-96  Radar  System  installed  in  E-2B  aircraft.  The  reliability 
improvements  from  these  equipment  improvements  are  also  addressed. 


Clyman,  Milton,  and  Grenetz.  Philip  (Information  Spectrum  Inc.).  Mainte¬ 
nance  Improvement:  An  Analysis  Approach  Including  Inferential  Techniques. 
Vol.  I.  ISI-W-7958-02A .  15  March  1979.  UNCLASSIFIED 

This  final  report  (in  four  volumes)  presents  the  results  of  research 
into  assessing  the  economic  impact  of  potentially  avoidable  maintenance 
actions  for  selected  Naval  aircraft  subsystems.  Maintenance  actions  re¬ 
quiring  no  repair  and  those  resulting  in  induced  defects  and  failure-to- 
correct  are  identified.  A  coarse  evaluation  of  BIT  effectiveness  is  made. 


Cook.  Thomas  N.  (Sikorsky  Aircraft),  and  Arlano.  John  (Applied  Technology 
Laboratory).  "Analysis  of  Fault-Isolation  Criteria/Techniques, "  1980 
Proceedings.  Annual  Reliability  and  Maintainability  Symposium,  pg.  29. 

This  paper  documents  an  investigation  by  Sikorsky  of  fault-isolation 
criteria  and  techniques  related  to  Army  aviation.  The  investigation  con¬ 
firmed  that  fault-isolation  maintenance  is  a  significant  factor  in  the 
cost  of  operating  Army  helicopters.  The  most  frequent  criticism  voiced  by 
Army  personnel  in  the  field  concerns  the  generally  poor  quality  of  trouble¬ 
shooting  data  in  maintenance  publications.  Fault-isolation  analysis  tech¬ 
nique  (FIAT)  was  developed  to  facilitate  the  identification  of  symptom/ 
cause  relationships  and  the  collection,  processing,  and  organization  of 
data  required  for  the  preparation  of  maintenance  manuals.  Cases  of  repet¬ 
itive  troubleshooting  on  the  aircraft  were  analyzed,  with  the  following 
conclusions: 

-  Slightly  more  than  one-fourth  of  the  symptoms  associated  with  non¬ 
avionics  systems  of  the  CH-54  involved  one  or  more  errors  in 
troubleshooting. 

-  An  improperly  performed  fault-isolation  task  occurred  approximately 
every  76  flight  hours. 

It  was  recognized,  however,  that  the  errors  in  fault  isolation  documented 
in  the  ORME  data  represented  only  a  fraction  of  the  errors  that  actually 
occurred. 


UNCLASSIFIED 


Army  aviation  fault-isolation  maintenance  was  investigated.  An  im¬ 
proved  approach  to  the  development  of  fault-isolation  maintenance  data  for 
complex  systems  was  developed.  This  approach  (FIAT)  facilitates  the  iden¬ 
tification  of  symptom/cause  relationships  and  the  collection,  processing, 
and  organization  of  data  required  for  the  preparation  of  maintenance 
manuals.  (Man-hours  and  costs  of  no-defect  maintenance  actions  are 
tabulated. ) 


Coppola,  Anthony  (Rome  Air  Development  Center).  A  Design  Guide  for  Built- 
In-Test  (BIT) .  RADC-78-224,  April  1979.  UNCLASSIFIED 

This  report  summarizes  available  information  for  use  in  designing 
built-in-test  (BIT)  capabilities  in  electronic  systems.  It  describes  the 
various  types  of  BIT.  design  considerations  and  examples,  data  used  in  BIT 
design,  display  options,  coupling  and  shielding  considerations,  and  opti¬ 
mization  models. 


Cummings,  Kathy,  and  Hardesty,  Walter  (Naval  Weapons  Center).  Automation 
of  a  Maintainability  Prediction,  NWC  TP  6198,  August  1980.  UNCLASSIFIED 

A  maintainability  prediction  is  automated  through  the  use  of  the 
Pascal  programming  language  on  the  UNIVAG  1110.  This  report  provides  the 
information  necessary  to  describe  the  process  for  automating  a  maintaina¬ 
bility  prediction.  It  also  Includes  instructions  necessary  for  operating 
the  maintainability  Prediction  Computer  Program.  (Does  not  use  RTOK  rates, 
FD,  or  FI.) 


Dussault,  Heather  B.  (Rome  Air  Development  Center).  The  Evolution  and 
Practical  Applications  of  Failure  Modes  and  Effects  Analyses.  RADC-TR- 
83-72.  March  1983.  UNCLASSIFIED 

This  report  is  intended  to  give  reader  a  broad,  general  background  in 
the  techniques  available  for  failure  effects  analysis  and  their  usefulness. 
Sixteen  separate  techniques,  ranging  from  tabular  failure  modes  and  effects 
analysis  and  fault  tree  analysis  to  newer  techniques  such  as  hardware/ 
software  interface  analysis,  are  discussed. 


Dynamics  Research  Corporation,  Systems  Division.  Program  Management  Review 
(PMR)  Minutes  (AFLC  WSMIS).  E-9296-U.  14  May  1984.  UNCLASSIFIED 

Among  other  things,  these  minutes  present  Information  about  the  Air 
Force  Equipment  Maintenance  Management  Information  Systems  (AFEMMIS) , 
including  the  Core  Automated  Maintenance  System  (CAMS),  the  Generic  Inte¬ 
grated  Maintenance  Diagnostics  System  (GIMADS).  and  the  Equipment  Mainte¬ 
nance  Data  Base  (EMDB) . 


Everly.  Lt.  Col.  Julian  R.  (U.S.  Army  War  College).  Maintenance  Quality 
Control:  A  Critical  Appraisal.  AD  A129757,  15  April  1983.  UNCLASSIFIED 

The  ambiguity  in  maintenance  Inspection  standards,  the  Inconsistency 
among  various  technical  Inspectors,  and  the  Impact  of  developmental  and 
fielding  policies  concerning  test  measurement  and  diagnostic  equipment 
(TMDE)  are  examined  relative  to  the  conduct  of  sound  quality  and  production 
management  practices.  An  alternative  approach  to  quality  management  is 
proposed  in  the  interpretation  and  application  of  sound  maintenance  stan¬ 
dards.  conduct  of  in-house  and  TRADOC  training  programs,  development  of 
TMDE  and  special  tools,  and  greater  utilization  of  warrant  officers  in  the 
role  of  quality  managers.  (No  definitions  provided.) 


Ferguson.  Capt.  Gerald  (Air  Force  Institute  of  Technology).  Aircraft 
Maintenance  Expert  Systems  (Master's  Thesis),  AFIT/GCS/EE/83D-9,  November 
1983 .  UNCLASSIFIED 

This  thesis  provides  design  considerations  for  implementation  of  an 
"expert  system"  to  assist  in  the  diagnosis  of  aircraft  problems.  It 
illustrates  the  characteristics  required  for  an  automated  diagnostic  sys¬ 
tem  to  assist  the  average  aircraft  technician  in  the  performance  of  his  or 
her  duties.  The  design  of  a  "knowledge  base"  and  "inference  procedure” 
for  such  a  system  are  presented.  A  working  system  model  was  developed  on 
a  microcomputer  to  demonstrate  the  feasibility  for  a  full-scale  mainte¬ 
nance  expert  system. 


Fiorentino.  Eugene  (Rome  Air  Development  Center).  The  Use  of  Air  Force 
Field  Maintenance  Data  for  R&M  Assessments  of  Ground  Electronic  Systems, 
RADC-TR-79-13,  April  1979.  UNCLASSIFIED 

The  R&M  estimates  derived  from  field  data  are  compared  with  similar 
estimates  obtained  from  the  R&M  design  and  development  program.  The  study 
identifies  the  major  limitations  in  the  field  data  for  use  in  field  R&M 
assessment.  Recommendations  for  improving  the  quality  and  usability  of 
the  field  data  are  also  made. 


Fleming.  Dr.  R.,  and  Dehoff,  Dr.  R.  (Systems  Control  Technology,  Inc). 
Turbine  Engine  Fault  Detection  and  Isolation  Program,  Vol  II:  Maintenance 
Model  Development.  AFWAL-TR-82-2058 ,  August  1982.  UNCLASSIFIED 

Maintenance  decision  analysis  models  for  evaluation  of  the  TF-34 
maintenance  process  are  formulated.  These  models  form  the  foundation  for 
the  U.S.  Air  Force  to  establish  techniques  for  determining  optimal  policy 
for  troubleshooting  and  maintenance  of  its  aircraft  engines  using  decision 
analysis  methods.  Model  structures  and  parameters  as  well  as  input  and 
output  are  treated. 


1C)  _  _ _ 

Organizational  Operating  Manual,  Maintenance  Management  Information  System 
for  Division  86.  Research  Note  84-18,  January  1984.  UNCLASSIFIED 

The  purpose  of  this  effort  is  to  develop  the  Maintenance  Performance 
System-Organizational  (MPS-O),  which  is  an  integrated  system  for  measuring 
maintenance  performance,  diagnosing  performance  problems,  taking  corrective 
actions,  and  providing  training.  This  report  provides  instructions  for 
operating  the  maintenance  management  information  systems  of  MPS-O.  (No 
definitions  provided.) 


Fuqa.  N.  (IIT  Research  Institute).  Electronic  Equipment  Maintainability 
Data,  EEMD-1 ,  Fall  1980.  UNCLASSIFIED 

This  is  the  first  of  a  series  of  maintainability  data  publications  at 
the  system/ equipment  level.  The  data  are  intended  to  complement  MIL-HDBK- 
217,  MIL-STD-883.  MIL-STD-785.  MIL-STD-470.  and  MIL-HDBK-472. 


Garfield,  J.  (Gould  NAVCOM  Division),  and  Razovsky,  I.  “Economical  Fault 
Isolation  Analysis."  1985  Proceedings,  Annual  Reliability  and  Maintaina¬ 
bility  Symposium,  pg.  480. 

This  paper  presents  a  rationale  for,  and  the  results  of,  a  tailoring 
process  applied  to  the  failure  analysis  described,  for  example,  in  MIL-STD- 
1629A.  MIL-STD-1543A.  and  ARP  926A.  which  is  used  for  fault  isolation  of 
shop  repairable  units  (SRUs). 


Gemas.  Capt.  G.  L.  (Department  of  Communication.  AFIT/LSH.  Wrlght-Patterson 
Air  Force  Base).  Aircraft  Avionics  System  Maintenance  Cannot  Duplicate 
and  Retest-OK  Analytical  Source  Analysis  (Master's  Thesis).  LSSR  49-83, 
September  1983.  UNCLASSIFIED 

This  study  focuses  on  the  aircraft  avionic  maintenance  problems  of 
cannot  duplicate  (CND)  and  retest-OK  (RTOK)  for  three  sampled  F-16  wings. 
Analytical  and  survey  methods  were  used  to  determine  causes  of  CND  and 
RTOK  occurrences. 


Gilmore.  Jordan,  and  Pisano  (General  Electric  Company,  Aircraft  Engine 
Group).  Assessment  of  Augmented  Electronic  Fuel  Controls  for  Modular 
Engine  Diagnostics  and  Condition  Monitoring,  USARTL-TR-78-32,  December 
1978.  UNCLASSIFIED 

Fault  isolation  to  the  module  and  LRU  levels  by  means  of  a  diagnostic 
and  condition  monitoring  (D&CM)  system  integrated  with  a  full -authority 
digital  electronic  control  (FADEC)  is  evaluated  in  this  study.  A  prelimi¬ 
nary  assessment  of  D&CM  system  parameters  required  for  performing  the 
diagnostic  functions  on  the  current  T-700  engine  is  also  included.  An 
integral  part  of  the  GE  FADEC  system  is  failure  indication  and  corrective 
action  (FICA)  based  on  extended  Kalman-Bucy  filtering  techniques. 


Gleason.  Cape.  Daniel  (Rome  Air  Development  Center).  "Analysis  of  Built- 
In-Test  Accuracy,"  1982  Proceedings,  Annual  Reliability  and  Maintainability 
Symposium,  pg.  370. 

Built-in-test  accuracy  is  a  combined  measure  of  fault-detection  capa¬ 
bility  and  false-alarm  occurrences.  This  paper  provides  a  Markovian  anal¬ 
ysis  of  BIT  accuracy.  The  results  of  the  analysis  are  used  to  develop 
trade-off  techniques  for  achieving  optimal  BIT  accuracy  levels  (gives 
probability  equations  of  false  alarm  and  failure  detections). 


Gold.  Kleine.  et  al.  (Xyzyx  Information  Corporation) .  Aircraft  Maintenance 
Effectiveness  Simulation  (AMES)  Model.  NAVTRAEQUIPC-77-D-0028-1 .  February 
1980 .  UNCLASSIFIED 

This  report  describes  a  project  to  develop  and  test  a  functional 
simulation  model  of  aircraft  maintenance.  The  model,  called  AMES,  measures 
the  effects  of  human  errors  in  maintenance. 


Gordon.  Capt.  John  S.  (TACC  AUTO  Engineering  Division).  An  Investigation 
of  Reliability,  Maintainability,  and  Availability  in  TACC  Automation  Pro¬ 
gram.  ESD-TR-79-139.  March  1979.  UNCLASSIFIED 

This  report  presents  RAM  information  (background  and  principles),  RAM 
complexities  and  methods  for  enhancing  RAM.  TACC  automation  RAM  complexi¬ 
ties  are  discussed  in  terms  of  definitions,  system  definition  deficiencies, 
and  RAM  predictions. 


Griffin.  Lt.  Col.  L.  D.  (Institute  for  Defense  Analysis),  and  Kern, 

George  A.  (Hughes  Aircraft  Company),  R&M  Parameter  Analysis  Document.  IDA 
D-27.  August  1983.  UNCLASSIFIED 

This  document  describes  the  four  categories  of  R&M  parameters  (readi¬ 
ness,  reliability,  maintainability,  and  manpower)  used  by  each  of  the 
services  (Army,  Navy,  Air  Force)  for  typical  military  systems  and  equip¬ 
ments.  In  addition  to  the  descriptions  of  each  parameter  (and  its  sub¬ 
sets).  parameter  strengths  and  weaknesses  are  discussed.  The  document 
includes  definitions  and  related  information. 


Grumman  Aerospace  Corporation.  Design  Guidelines  and  Optimization  Proce¬ 
dures  for  Test  Subsystem  Design,  RADC-TR-80-111,  April  1980.  UNCLASSIFIED 

Guidelines  and  procedures  for  optimizing  design  of  BIT  equipment  are 
provided.  Optimization  of  the  test  subsystem  is  achieved  by  properly 
specifying  three  key  design  parameters  (test  effectiveness,  mean  corrective 
maintenance  time,  and  test  subsystem  production  costs)  during  the  concep¬ 
tual  phase.  This  report  provides  mathematical  tools,  algorithms,  and 
trade-off  procedures  to  assist  the  designer  during  each  design  phase. 


when  final  design  data  are  available.  Predicted  parameters  Include  mean 
time  to  repair,  maximum  (percentile)  time  to  repair,  maintenance  man-hours 
per  repair,  and  fault-isolation  resolution.  The  technique  Includes  a  set 
of  time  standards  applicable  to  physical  maintenance  actions  associated 
with  current  construction  and  packaging  techniques. 


Hughes  Aircraft  Company.  RADC  Testability  Notebook.  RADC-TR-82-189,  June 
1982.  UNCLASSIFIED 

Inherent  testability  must  be  systematically  developed  and  integrated 
with  the  design  of  the  system,  and  the  requirements  for  testability  must 
be  accorded  the  same  level  of  recognition  as  performance,  reliability, 
maintainability,  availability,  supportablllty ,  and  safety.  This  testabil¬ 
ity  notebook  provides  fundamental  guidance  for  systematic  establishment 
throughout  the  development  cycle  of  the  requisite  inherent  testability  and 
comprehensive  testability  of  the  test-resource  mix  in  combination  with  the 
prime  system  design. 


Hughes  Aircraft  Company  tudv  of  the  Causes  of  Unnecessary  Removals  of 
Avionic  Equipment.  RADC-TR-83-2 ,  January  1983.  UNCLASSIFIED 

This  study  investigated  and  verified  the  causes  of  unnecessary  re¬ 
movals  of  suspect  items  from  selected  avionic  equipment.  The  report 
recommends  actions  that  are  useful  in  minimizing  unnecessary  removals. 


I IT  Research  Institute/Rel lability  Analysis  Center.  Correlation  of  Field 
Data  with  Reliability  Prediction  Models.  RADC-TR-81-329,  November  1981. 
UNCLASSIFIED 

This  report  considers  the  factors  influencing  the  goodness  of  fit  of 
MIL-HDBK-217C  prediction  models.  Areas  in  which  the  models  are  deficient 
are  identified  and  quantified.  Where  positive  inferences  are  possible,  a 
range  of  statistical  methods  are  used  to  give  an  unbiased  assessment.  The 
underlying  distribution  of  time  to  failure  is  investigated,  since  MIL-HDBK 
217C  assumes  a  constant-failure-rate  model. 


I IT  Research  Institute.  Least  Cost  Test  Profile,  Vol.  I,  RADC-TR-82-84 , 
April  1982.  UNCLASSIFIED 

This  study  developed  test  profiles  for  rigid-wall  tactical  shelters. 
Test  cost  data  and  test  results  were  obtained,  and  an  effort  to  determine 
the  correct  text  sequences  was  instituted.  The  operational  data,  test 
costs,  test  results,  and  output  from  the  test  sequence  effort  were  used  to 
develop  test  profiles  for  eight  members  of  the  standard  family  of  shelters 
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IIT  Research  Institute.  Least  Cost  Test  Profile.  Vol.  II,  RADC-TR-82-84 , 
April  1982.  UNCLASSIFIED 

This  study  developed  test  profiles  for  rigid-wall  tactical  shelters. 
Test  cost  data  and  test  results  were  obtained,  and  an  effort  to  determine 
the  correct  test  sequences  was  instituted.  The  operational  data,  test 
costs,  test  results,  and  output  from  the  test  sequence  effort  were  used  to 
develop  test  profiles  for  eight  members  of  the  standard  family  of  shelters. 


Institute  for  Defense  Analysis.  Built-in-Test  Equipment  Requirements 
Workshop.  Paper  P-1600.  August  1981.  UNCLASSIFIED 

A  workshop  was  held  for  the  purpose  of  assessing  progress  and  problems 
in  specifying  and  testing  BIT  used  in  complex  electronic  equipment.  The 
recommendation  is  that  the  current  specification  and  test  approach  be 
broadened  to  include  all  capabilities  associated  with  the  detection  and 
isolation  of  faults.  The  report  emphasizes  BIT  specification,  testing, 
and  evaluation. 


Kiener,  William  L..  and  Coppola,  Anthony.  "Joint  Services  Program  in 
Design  for  Testability,”  1981  Proceedings,  Annual  Reliability  and 
Maintainability  Symposium,  pg.  268. 

This  paper  is  divided  into  three  parts.  The  first  part  is  an  overview 
of  testability,  including  its  definition,  its  relationship  to  reliability 
and  maintainability,  and  its  importance  in  supporting  complex  weapon  sys¬ 
tems.  The  second  part  is  a  summary  of  Navy  subtasks  under  the  JLC  DFT 
program.  The  third  part  is  a  summary  of  the  Air  Force  testability  program. 


Knalzuk.  J.  (Syracuse  University).  Manual  Fault  Detection  Test  Set  Mini¬ 
mization.  RADC-TR-77-149.  May  1977.  UNCLASSIFIED 

This  report  describes  a  manual  procedure  for  minimizing  the  number  of 
tests  necessary  to  detect  a  single  "stuck  at”  fault  in  a  large-scale  inte¬ 
grated  circuit. 


Krause.  George  S.,  Jr.  (Westlnghouse  Electric  Corporation).  "Distributed 
Versus  Centralized  BIT/FIT  Processing,"  1985  Proceedings,  Annual  Relia¬ 
bility  and  Maintainability  Symposium,  pg.  291. 

This  paper  describes  two  system  testability  concepts  for  electronic 
systems.  The  first  is  centralized  BIT/FIT  processing  (for  fault  detection 
and  isolation  testing),  and  the  second  is  distributed  BIT/FIT  processing. 
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Lahore,  H.  (Boeing  Aerospace),  and  Gozzo,  L.  (RADC) .  “Artificial  Intelli¬ 
gence  Applications  to  Testability,"  1985  Proceedings,  Annual  Reliability 
and  Maintainability  Symposium,  pg.  276. 

i  This  study  provides  a  foundation  for  applying  AI  to  electronic  testa¬ 

bility  for  the  military.  Applications  Include  system-level  maintenance 
j  expert  and  smart  maintenance  expert  in  order  to  reduce  false  alarms  and 

j  Improve  fault  isolation  and  detection. 


Liu.  Nakajima,  Olivier,  et  al.  (Texas  Tech  University).  Nonlinear  Fault 
Diagnosis.  AD-A101053,  May  1981.  UNCLASSIFIED 

Several  research  projects  in  nonlinear  fault  diagnosis  are  summarized. 
Alternative  algorithms  for  the  solution  of  the  nonlinear  fault-diagnosis 
problem  are  presented,  together  with  a  dlagnoslblllty  theory  and  a  set  of 
criteria  that  an  “ideal"  fault -diagnosis  problem  should  strive  to  meet. 

(No  definitions  --  only  mathematical  theory.) 


Lockheed  California  Company.  Built-In-Test  and  External  Tester  Reliability 
Characteristics.  RADC-TR-80-32.  March  1980.  UNCLASSIFIED 

This  report  presents  the  results  of  a  study  of  the  reliability  impact 
of  BIT  and  external  test  equipment  on  prime  equipment  design  and  mainte¬ 
nance  downtime.  Sixty  units  were  analyzed  from  the  S-3A,  C-5A.  and  Mk  86 
weapon  systems  to  develop  BIT  and  external  tester  measurement  effectiveness 
versus  unit  design  characteristics.  Trade-off  criteria  were  developed  for 
predicting  BIT  and  test -equipment  reliability  during  the  acquisition  of 
new  systems. 


Malcolm.  J.  G.  (Hughes  Aircraft  Company).  "BIT  False  Alarms:  An  Important 
Factor  in  Operational  Readiness,"  1982  Proceedings,  Annual  Reliability  and 
Maintainability  Symposium,  pg.  206. 

The  premise  of  this  paper  is  that  current  avionic  systems  are  inher¬ 
ently  reliable  and  potentially  maintainable  at  high  rates  of  operational 
readiness.  This  paper  describes  some  root  causes  of  false  alarms  and 
identifies  solution  approaches,  including  improved  specifications,  improved 
analysis  techniques,  and  expanded  use  of  new  technology.  Resolving  the 
false-alarm  problem  can  result  in  a  major  improvement  in  operational 
readiness. 


Malcom.  J.  G.  (Hughes  Aircraft  Company).  "Practical  Applications  of  Bayes’ 
Formulas."  1983  Proceedings,  Annual  Reliability  and  Maintainability 
Symposium,  pg.  180. 

Five  practical  applications  of  Bayes'  formulas  are  presented.  One  of 
these  deals  with  the  problem  that  the  military  has  been  wrestling  with  for 
years,  namely,  the  excessive  number  of  units  checking  no  fault  in  the 


Intermediate-level  shop.  This  problem  Is  variously  known  as  the  retest-OK 
(RTOK)  problem  or  the  bench  check  serviceable  (BCS)  problem  or  the  cannot 
duplicate  (CND)  problem.  Traditionally,  this  problem  is  attacked  as  though 
It  represents  a  deficiency  in  the  test  that  was  used  to  cause  the  unit  to 
be  removed  and  replaced  In  the  system,  typically  built-in  test  (BIT). 

This  paper  demonstrates  that  the  RTOK  rate  can  be  a  function  of  the  preva¬ 
lence  of  faulty  systems.  Statistical  guidelines  are  presented  in  the 
paper.  Indicating  how  designers  can  develop  tests  that  will  provide  an 
optimal  balance  between  false  alarms  and  missed  faults. 


McWhirter,  Johnson,  and  McLane  (David  W.  Taylor  Naval  Ship  R&D  Center). 

A  Shipboard  Machinery  Performance  Monitoring  System  Concept.  DTNSRDC/ PAS- 
78/30.  February  1979.  UNCLASSIFIED 

This  report  describes  a  concept  for  an  instrumentation  and  monitoring 
system  for  Naval  shipboard  monitoring.  Specific  topics  addressed  In  this 
report  include  system  capability  requirements,  data  collection,  and  local 
remote  processing.  This  system  Is  being  developed  to  enable  shipboard 
personnel  to  predict  maintenance  action,  to  reduce  maintenance  time  re¬ 
quired.  and  to  provide  a  tool  for  maintenance  management.  (NO  definitions 
provided . ) 


Miehle.  William,  and  Siegel.  Arthur  (Applied  Psychological  Services). 
Development  of  Performance  Evaluative  Measures.  December  1967. 

UNCLASSIFIED 

The  logic  of  a  technique  for  employing  technician  "confidence  that  a 
defect  exists"  for  maximizing  the  probability  of  malfunction  recognition 
is  described.  Operator  characteristics  curves  are  derived  for  a  variety 
of  distributions  of  “confidence."  The  implications  of  the  work  for  train¬ 
ing  and  post-training  performance  evaluation  are  pointed  out.  Probabil¬ 
ities  of  saying  "yes"  or  "no"  when  there  is  a  defect  or  saying  "yes"  or 
"no"  when  there  is  no  defect  are  given.  (No  definitions,  but  numerous 
formulas. ) 


MIL-HDBK-217D .  Reliability  Prediction  of  Electronic  Equipment. 

9  April  1979. 

This  handbook  establishes  uniform  methods  for  predicting  the  relia¬ 
bility  of  military  electronic  equipment  and  systems.  It  provides  a  common 
basis  for  reliability  predictions  during  acquisition  programs  for  military 
electronic  systems  and  equipment.  It  establishes  a  common  basis  for  com¬ 
paring  and  evaluating  reliability  predictions  of  related  or  competitive 
designs.  Two  methods  of  reliability  prediction  are  used  —  part  stress 
analysis  and  parts  count. 
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The  purpose  of  this  handbook  Is  to  familiarize  managers  and  design 
engineers  with  current  maintainability  prediction  procedures.  Four  pro* 
cedures  are  described. 


MIL-STD-471A. 
27  March  1973. 


Maintainability  Verlflcatlon/Demonstratlon/Evaluatlon. 


The  purpose  of  this  standard  is  to  establish  uniform  procedures,  test 
methods,  and  requirements  for  verification,  demonstration,  and  evaluation 
of  the  achievement  of  specified  maintainability  requirements  and  for 
assessment  of  the  Impact  of  planned  logistics  support.  The  standard 
Includes  lengthy  discussion  (In  addendum)  on  procedures  for  evaluation  and 
demonstration  of  equipment /system  BIT  and  ETE  fault  Isolation  and  testa¬ 
bility  attributes  (fault  isolation/detection),  but  no  specific  definitions 
can  be  derived  from  the  context  of  the  entire  discussion. 


MIL-STD- 1 309B .  Definitions  of  Terms  for  Test,  Measurement,  and  Diaqnostic 


Equipment ,  30  May  1975. 

Test,  measurement,  and  diagnostic  equipment  key  terms  are  defined  in 
order  to  improve  communications  and  to  facilitate  coordination.  Defini¬ 
tions  include  fault,  false  alarm,  fault  detection,  fault  Isolation,  and 
organizational  maintenance. 

MIL-STD- 1591.  Analvsis/Svnthesls  of  On-Aircraft  Fault  Diagnosis 
Subsystems ,  3  January  1977. 

This  standard  establishes  uniform  criteria  for  conducting  trade 
studies  to  determine  the  optimal  design  for  an  on-aircraft  fault-diagnosis/ 
Isolation  system,  referred  to  as  the  On-Board  Built-In  Test  (OBBIT)  System. 
The  standard  Is  applicable  to  DoD  procurements  that  Include  the  development 
of  on-aircraft  fault-diagnosis/isolation  systems  where  a  selection  can  be 
made  between  such  alternatives  as  central  computer  controlled,  on-board 
centrally  polled  built-in  test  equipment  (BITE),  decentralized  BITE, 
detached  aerospace  ground  equipment  (AGE) ,  or  combinations  of  the 
preceding. 


MIL-STD-2076 (AS) .  General  Requirements  for  Unit  Under  Test  Compatibility 
with  Automatic  Test  Equipment.  1  March  1978. 


MIL-STD-2077 (AS) .  General  Requirements  for  Test  Program  Sets. 

15  July  1975. 

This  standard  establishes  the  requirements  for  the  development,  test 
documentation,  configuration  management,  quality  assurance,  and  preparation 
for  delivery  of  test  programs  (TPs)  and  related  hardware  and  documentation 
to  be  used  in  conjunction  with  an  appropriate  ATE  to  test  IJUTs. 


MIL-STD-2165 .  Testability  Program  for  Electronic  Systems  and  Equipments. 

26  January  1985 . 

This  standard  prescribes  a  uniform  approach  to  testability  program 
planning,  establishment  of  testability  (including  BIT)  requirements,  test¬ 
ability  analysis,  prediction  and  evaluation,  and  preparation  of  testability 
documentation.  It  includes  testability  program  planning,  testability 
requirements,  testability  design,  testability  prediction,  testability 
demonstration,  testability  data  collection  and  analysis,  documentation  of 
testability  programs,  and  testability  review. 


Mulligan,  Joseph  (Management  and  Technical  Services  Company  [Air 
Force] ) .  Logic  Tree  Troubleshooting  Aid:  Organizational  and  Intermediate 
Maintenance ,  AFHRL-TR-79-49 ,  January  1980.  UNCLASSIFIED 

This  report  provides  a  draft  military  specification  for  use  in  the 
procurement  of  logic  tree  troubleshooting  aids  (LTTAs) .  A  thorough  review 
of  the  state  of  the  art  in  preparing  and  using  LTTAs  was  made  to  provide 
the  basis  for  developing  the  draft  specification.  The  draft  specification 
provides  specific  and  general  requirements  for  the  development  of  LTTAs, 
including  task  analysis  and  development  of  troubleshooting  procedures  and 
check-out  procedures.  (No  definitions  provided.) 


NAVMATINST  3960. 6B.  Procedures  for  Acguisition  Category  IV  Test  and 
Evaluation  Plans  (TEPS) ,  9  February  1981. 

This  instruction  applies  to  all  Navy  Acquisition  Category  I,  II,  III, 
and  IV  programs  except  nuclear  weapon  subsystems  and  nuclear  propulsion 
subsystems.  Product  assessment  through  tests  and  evaluations,  including 
early  participation  by  the  commander,  is  seen  as  a  realistic  method  of 
ensuring  that  new  systems  will  be  operationally  effective  and  suitable 
before  being  approved  for  service  use. 


Neuman,  George  W.  (Giordano  Associates,  Inc.).  Testing  Technology  Working 
Group  Report  (IDA/OSD  R&M  Study) .  IDA  D-41,  August  1983.  UNCLASSIFIED 


This  study  report  addresses  the  requirements  for  a  testing  technology 
development  program.  The  study  is  part  of  a  larger  Reliability  and  Main¬ 
tainability  Improvement  Study  Program.  The  first  portion  of  this  report 


describes  the  entire  study  and  how  testing  technology  fits  Into  Its  frame¬ 
work.  This  Is  followed  by  a  description  of  the  problem,  scope,  goals, 
objectives,  approach,  content,  payoffs,  conclusions,  and  recommendations 
related  to  a  testing  technology  program.  (Addresses  false-alarm-rate 
requirements  but  provides  no  definitions.) 


Osborn.  Jack  (Institute  for  Defense  Analysis).  CAD/CAM  Technology  Working 
Group  Report  (IDA/OSD  R&M  Study) ,  AD-A137761.  August  1983.  UNCLASSIFIED 

The  goal  of  the  report  Is  to  identify  means  by  which  corapu ter -aided 
technologies  can  lead  to  quantum  improvements  in  R&M.  The  report  articu¬ 
lates  a  model  of  the  process  of  taking  a  weapon  system  from  concept  to 
product  using  computer-aided  technologies.  Two  major  Issues  concerning 
these  technologies  are  developed:  effective  application  of  existing 
computer-aided  technologies  and  communications  among  subsets  of  computer- 
aided  technologies,  e.g..  CAE,  CAD.  CAM.  (No  definitions  provided.) 


Ramirez,  Miguel  A.  (Westlnghouse  Electric  Corporation).  "Achieving  Main¬ 
tainability  by  Random  Fault  Injection."  1982  Proceedings,  Annual  Relia¬ 
bility  and  Maintainability  Symposium,  pg.  291. 

This  paper  describes  a  random  fault-injection  testing  technique  that, 
when  implemented,  will  significantly  Improve  the  probability  of  meeting 
maintainability  requirements  in  the  field.  The  proposed  random  fault- 
injection  technique  provides  a  testability-growth  program  that  concentrates 
on  fault-detectlon/lsolatlon  effectiveness  and  mean  time  to  repair  (MTTR) . 


Rouse.  Rouse,  Hunt,  et  al.  (Coordinated  Science  Laboratory,  College  of 
Engineering) .  Human  Decision-Making  In  Computer-Aided  Fault  Diagnosis. 

TR  434.  January  1980.  UNCLASSIFIED 

This  report  summarizes  six  experiments  conducted  to  Increase  our 
understanding  of  human  performance  on  diagnostic  tasks  and,  in  the  process, 
to  Investigate  the  feasibility  of  using  context-free  computer-based  simu¬ 
lations  to  Improve  troubleshooting  skills.  Results  provide  a  data  base 
for  both  theoretical  Issues  In  fault  diagnosis  and  practical  application 
of  computer  aiding  to  live  system  performance.  (No  definitions  provided.) 


Slevers.  Kravetz.  Dussia,  and  Jackson  (Fail-Safe  Technology  Corporation). 
Military  Standard  Fault-Tolerant  Microcomputer.  FR-OOlAD,  July  1982. 
UNCLASSIFIED 

This  report  includes  the  results  of  a  feasibility  study,  preliminary 
design,  and  recommendations  for  subsequent  work.  Fault  tolerance  is  the 
unique  attribute  of  a  computer  system  that  enables  that  system  to  continue 
its  program-specified  behavior  in  spite  of  the  occurrence  of  faults.  The 
computer  system  described  In  this  report  provides  fault  detection.  Isola¬ 
tion.  and  repair.  An  overview  of  the  state-of-the-art  concepts  and  tech¬ 
niques  employed  in  fault -tolerant  computer  designs  Is  added. 


Distributed  Systems.  RADC-TR-83-36 ,  February  1983.  UNCLASSIFIED 

The  objective  of  this  study  Is  to  provide  a  foundation  for  the  devel¬ 
opment  of  design  measures  and  guidelines  for  the  design  of  fault-tolerant 
systems.  Taxonomies  of  fault  tolerance  and  distributed  systems  are  devel¬ 
oped.  and  typical  Air  Force  C3I  needs  in  both  fault-tolerant  and  distri¬ 
buted  computer  systems  are  characterized.  Key  issues  in  the  design  of 
fault-tolerant  distributed  systems  are  identified.  Fault-location  tech¬ 
niques  for  specific  computer  configurations  found  in  C3I  applications 
are  detailed. 


Sperry  Corporation.  Design  Guide,  Bullt-In-Test  (BIT),  and  Built-In-Test 
Equipment  (BITE)  for  Army  Missile  Systems,  TR-RL-CR-81-4,  11  April  1981. 
UNCLASSIFIED 

This  report  documents  the  first  draft  of  a  design  guide  and  has  been 
prepared  as  an  aid  to  the  project  manager,  beginning  with  the  conceptual 
phase  through  development,  and  as  a  guide  to  the  system  design  engineer 


concerned  with  the  incorporation  of  built-in-test  (BIT)  and  built-in-test 
equipment  (BITE)  into  the  weapon  system.  It  is  not  the  Intent  of  this 
document  to  detail  the  “how  to"  but  rather  to  identify  those  subject  areas 
that  need  to  be  considered  in  determining  the  requirement  for  BIT.  Uses 
MIL -STD- 1309. 


Stander,  Carvel  R.  (The  Boeing  Company).  “Fault  Isolation  BITE  for  In¬ 
creased  Productivity,"  1982  Proceedings.  Annual  Reliability  and  Maintaina¬ 
bility  Symposium,  pg.  365. 

With  digital  avionics  comes  a  quantum  leap  in  system  complexity  and 
the  need  for  a  comparable  increase  in  fault-isolation  ability,  with  its 
related  ramifications.  The  Boeing  Company  began  early  in  its  new  airplane 
design  to  study  the  fault-isolation  problem.  From  the  studies  came  new 
design  criteria,  numerical  objectives,  and  verification  methods.  The  re¬ 
sults  produced  major  gains  in  fault-isolation  capability  on  complex  digital 
systems  through  the  development  of  a  new  generation  of  BITE.  The  new  757/ 
767  BITE  is  designed  for  the  mechanic,  not  the  engineer.  It  eliminates 
many  of  the  existing  problems  with  today's  BITE,  such  as  the  inability  to 
deal  with  intermittent  faults. 


Towne.  Douglas  M.;  Johnson.  Mark;  and  Corwin,  William  (Behavioral  Technol¬ 
ogy  Laboratory).  A  Performance-Based  Technique  for  Assessing  Equipment 
Maintainability,  AD-A1 33518.  August  1983.  UNCLASSIFIED 

Maintainability  projections  were  made  for  a  digital  inferred 
transmit ter/ receiver  system,  specially  constructed  to  be  configured  in  two 
functionally  equivalent  forms.  Technicians  worked  to  identify  and  resolve 
eight  Inserted  malfunctions  each,  using  built-in  indicators  and  standard 
test  equipment.  A  measure  of  design  complexity  is  proposed  for  the  evalu¬ 
ation  of  maintainability.  This  measure,  mean  number  of  indicators  neces¬ 
sary  to  accomplish  fault  Isolation,  is  sensitive  to  multiplicity  of  fault 
modes  and  to  the  extent  to  which  fault  symptoms  are  confounded  at  the 
malntalner  Interface.  (No  definitions  provided.) 


Wardell,  Howard  (Directorate  of  Training  Developments).  Final  Independent 
Evaluation  Plan  for  XM1  Turret  Organizational  Maintenance  Trainer,  TRADOC 
ACN  39377.  March  1981.  UNCLASSIFIED 

The  major  objectives  of  this  Independent  Evaluation  Plan  (IEP)  are  to 
determine  the  effectiveness  of  XMl  TOMT,  the  logistics  supportability  of 
the  XMl  TOMT.  and  the  adequacy  of  the  XMl  TOMT  human-factors  design  with 
respect  to  operability  and  acceptability.  Appendix  A  has  failure 
definitions  —  what  does  and  does  not  constitute  a  failure  —  but  no  other 
definitions. 


Weapon  System  Effectiveness  Industry  Advisory  Corani ttee.  Chairman's  Pinal 
Report .  AFSC-TR-65-6 .  January  1965.  UNCLASSIFIED 

The  weapon  System  Effectiveness  Industry  Advisory  Committee  (WSEIAC) 
developed  a  measure  of  the  extent  to  which  a  system  may  be  expected  to 
achieve  a  set  of  specific  mission  requirements.  The  WSEIAC  approach  Is 
based  on  the  concept  of  system-state  occurrence  probabilities  and  system- 
state  capabilities  for  performing  the  mission.  A  system  state  is  a 
distinguishable  condition  of  the  system  that  results  from  events  occurring 
prior  to  and  during  the  mission. 

Wells,  C.  F.  (ARINC  Research  Corporation).  Organizational  Maintenance 
Requirements  for  Augmenting  Fault-Isolation  Procedures  for  P-3C  Avionics, 


928-02-3-1049,  May  1970.  UNCLASSIFIED 

This  report  describes  the  avionic  systems  of  the  P-3C  aircraft  for 
which  modules  are  needed  in  troubleshooting  by  the  substitution  method, 
the  modules  needed  to  outfit  a  module  caddy,  alternative  maintenance  con¬ 
cepts.  and  the  effect  of  using  initially  provisioned  spares  for  outfitting 
module  caddies.  (No  definitions  provided.) 
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Abnormal  Fault  Isolation  -  Techniques  used  to  Identify  the  cause  of  SUT 
failure  by  means  other  than  normal  system  maintenance  procedures.  For 
example:  (1)  removal  of  multiple  replaceable  units.  (2)  shotgun  removal 
of  replaceable  units  until  the  SUT  Is  operational. 

AFTO  349  -  Air  Force  Maintenance  Data  Collection  Record. 

Bad  Actor  -  Any  SUT  with  repeat  failure  Indications  that  cannot  be  dupli¬ 
cated  or  verified  during  normal  system  maintenance.  Bad  actors  may  be 
"recognized"  over  a  period  of  time  or  may  be  “Indicated"  by  outside 
sources.  Bad  actors  may  be  generic  (e.g.,  LRU  type)  or  specific  (i.e.,  a 
given  serial  number). 

Cannot  Duplicate  (CND)  or  No  Fault  Found  (NFF)  -  There  is  a  prior  indica¬ 
tion  of  failure  and  the  failure  cannot  be  duplicated  by  maintenance. 

False-Alarm  Hate  (FAR)  -  The  rate  of  occurrence  of  false  alarms,  typically 
computed  as  the  time-normalized  sum  of  false  alarms,  where  the  time 
normalized  Is  either  calendar  or  operating  hours. 

Intermittent  Failure  -  Transient  failure  mode  of  the  SUT  that  is  not  repro¬ 
ducible  by  using  normal  system  maintenance.  The  failure  may  or  may  not  be 
present  during  maintenance  checks.  Repeat  transient  failures  may  label  an 
SUT  as  a  “bad  actor"  and  result  in  replacement  without  maintenance  verifi¬ 
cation  of  the  fault. 

Maintenance  System  Fault  Detection  -  An  Indication  is  provided  by  normal 
system  maintenance  that  the  SUT  is  not  functioning  properly  because  of  a 
real  failure  within  the  SUT.  Fault  detection  may  be  subdivided  into  the 
following  categories:  BITE  fault  detection,  automatic  fault  detection, 
and  manual  or  semiautomatic  fault  detection. 

Maintenance  System  Fault  Isolation  -  Ability  to  identify  all  failed  re¬ 
placeable  units  within  the  SUT  using  normal  system  maintenance.  Fault 
Isolation  may  be  subdivided  Into  the  following  categories:  BIT  fault  iso¬ 
lation.  automatic  fault  Isolation,  and  manual  or  semiautomatic  fault 
Isolation. 


Nonrelevant  Event  -  Any  fault  Indication  that  does  not  result  in  a  mainte¬ 
nance  action. 

Normal  System  Maintenance  (NSM)  -  Techniques  that  are  specified  as  standard 
operating  procedures  for  use  of  BIT,  ATE.  semiautomatic,  or  documented 
manual  detection  and  troubleshooting  for  a  given  system  under  test.  This 
Includes  regular  calendar  checks  and  normal  go-checks.  It  is  sometimes 
called  "defined  means." 

Not  Isolatable  This  Station  (NITS)  -  Normal  or  abnormal  fault-isolation 
procedures  cannot  determine  the  cause  of  fault  in  the  SUT.  Maintenance 
concept  at  O-level  may  be  to  ship  the  SUT  to  another  level. 

Redball  -  A  last-ditch  effort  to  save  a  mission  when  the  scheduled  aircraft 
is  faulty.  TAC  and  SAC  call  this  "Redball."  and  MAC  calls  it  "Red  Streak." 
It  has  additionally  been  referred  to  as  "Blue  Streak"  by  SAC. 

Retest-OK  (RTOK)  -  A  replaceable  unit  is  removed,  but  no  failure  is 
discovered  at  subsequent  levels  of  maintenance.  A  RTOK  does  not 
automatically  imply  that  no  failure  exists. 

Shotgun  Maintenance  -  Random  removal  and  replacement  of  LRUs  in  order  to 
find  and  repair  faults. 

Subsystem  False  Alarm  -  A  failure  indication  in  a  subsystem  when  there  is 
no  failure  in  the  system. 

Subsystem  Improper  Fault  Detection  -  Fault  is  within  the  subsystem  other 
than  the  one  in  which  detection  occurs. 

Subsystem  Improper  Fault  Isolation  -  All  but  not  only  failed  units  are 
Isolated. 

Subsystem  Proper  Fault  Detection  -  Fault  is  within  the  subsystem  in  which 
detection  occurs. 

Subsystem  Proper  Fault  Isolation  -  Only  and  all  failed  units  are  isolated. 

Subsystem  Under  Test  (SUT)  -  All  of  the  equipment  associated  with  a  sub¬ 
system.  including  BITE  but  excluding  test  equipment  that  is  not  physically 
attached  during  normal  operation. 

System  False  Alarm  -  Normal  system  maintenance  indicates  a  failure  in  the 
SUT  when  there  is  no  failure  present. 

System  Go-Check  -  Normal  maintenance  procedure  used  to  verify  that  SUT  is 
functioning  properly. 

System  Improper  Fault  Isolation  -  All  but  not  only  failed  units  are 
Isolated. 

System  Proper  Fault  Isolation  -  Only  and  all  failed  units  are  isolated. 


ACRONYMS  AND  SYMBOLS 


AFEMMIS 

Air  Force  Equipment  Maintenance  Management  Information 

System 

AFTO 

Air  Force  Technical  Order 

AGS 

Aircraft  Generation  Squadron 

ANG 

Air  National  Guard 

AT 

Action  taken 

ATE 

Automatic  test  equipment 

BCS 

Bench  check  serviceable 

BIT 

Built-in-test 

BITE 

Built-in-test  equipment 

CAMS 

Core  Automated  Maintenance  System 

CFD 

Component  feedback  dominance 

CND 

Cannot  duplicate 

CNDj 

Cannot  Duplicate  events  of  real  failures 

CND 

o 

Operator-induced  cannot  duplicate 

CNDp 

Cannot  duplicate  events  of  false  alarms 

CND 

s 

Cannot  duplicate  results  of  NSM-generated  maintenance  actions 

CND_ 

T 

Improper  cannot  duplicate 

CNF 

Cannot  find 

CRS 

Component  Repair  Squadron 

D&CM 

Diagnostic  and  condition  monitoring 

DFT 

Design  for  testability 

DP 

Degree  of  parallelism 

a 

E 

Conditional  effectiveness 

ED 

External  dependency 

EDF 

External  dependency  factor 

EMDB 

Equipment  maintenance  data  base 

ETE 

Electronic  test  equipment 

EW 

Electronic  warfare 

B-87 
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F 

FA 

FAR 

FCND 

FD 

F°s 

FFA 

FFAj 


FFASS 
FFA. 


ssc 


FFD 

FFD 

C 

FFD 

C 

FFl" 

FFI£ 

FFIc 

FFI 


ss 


ssc 


ss 


ssc 


FI 

FI. 


FIs 

FISS 

FIu 

FIAT 

FICA 

FL 

FMECA 

FOM 


Maintenance  term  for  failures 
False-alarms 

Operator-induced  false  alarms 
False-alarm  rate 

Fraction  of  maintenance  actions  that  result  in  CND 
Maintenance  terra  for  valid  detection 
Maintenance  terra  for  valid  detections  by  operators 

Maintenance  terra  for  valid  detections  by  normal  system 
maintenance 

Fraction  of  false  alarms 

Fraction  of  false  alarms  that  measures  the  contribution  of 
the  jcn  component  to  FFA  (e.g.,  system/ operator/BIT) 
Fraction  of  false  alarms  in  the  subsystem 
Subsystem  contribution  of  fraction  of  false  alarms 

Fraction  of  faults  detected 

Fraction  of  faults  detected  in  the  subsystem 

Subsystem  contribution  of  fraction  of  faults  detected 

Fraction  of  faults  isolated 

Fraction  of  faults  isolated  performance 

Fraction  of  faults  isolated  in  the  subsystem 

Subsystem  contribution  of  fraction  of  faults  isolated 

Maintenance  term  for  faults  isolated  and  repaired 
Maintenance  term  for  faults  isolated  and  repaired  not  by 
normal  system  maintenance 

Fault  isolation  (of  real  failures)  by  normal  system 
maintenance 

Faults  Isolated  and  repaired  in  the  subsystem 
Fault  isolation  and  repair  of  nonfailures  (unnecessary 
repal r ) 

Fault-isolation  analysis  technique 
Failure  identification  and  corrective  action 
Feedback  loop 

Failure  modes  and  effects  criticality  analysis 
Figure  of  merit 


GIMADS  Generic  Integrated  Maintenance  Diagnostics  System 


How  Mai 
HW 


How  malfunctioned  (how  did  the  system/subsystem 
malfunction?) 

Hardware 


INS 

I/O 


Inertial  navigation  system 
Input/output 


s 

SAC 

SPO 

SRU 

SS 

ssc 

ST 

STAMP 

SUT 

SW 


System  survivability 
Strategic  Air  Command 
System  Program  Office 
Shop  replaceable  unit 
Subsystem 

Subsystem  contribution 
Self-test 

System  Testability  and  Maintenance  Program 

Subsystem  under  test 

Software 


TAC  Tactical  Air  Command 

TE  Test  equipment 

TMDE  Test  measurement  and  diagnostic  equipment 

TP  Test  program 

TPg  Normalized  test  point 

U  Maintenance  term  for  undetected  failures  (not  measurable) 

W/G  Warranty/guarantee 

WSEIAC  Weapon  System  Effectiveness  Industry  Advisory  Committee 

WSMIS  Weapon  System  Management  Information  System 

p  Empirical  coefficient  that  represents  the  percentage  of 

total  generated  CND  that  are  false  alarms 

0s  Empirical  coefficient  that  represents  the  percentage 

of  NSM-generated  CND  values  to  false  alarms 

Yj  Allocates  the  portion  of  the  subsystem  contribution  to 

detection  that  applies  to  the  system 

Aj  Allocates  the  portion  of  the  subsystem  contribution  to 

false  alarm  that  carries  forward  to  the  system 

n  Flow  model  functional 

t  Small  time  increment  or  temporary  variable 

Represents  the  cross-detection  of  subsystem  J  as  a  result 
of  failures  in  subsystem  i  and  t  ^  »  0 

Q  Population  operation  that  enumerates  the  membership  of  a 

set 

%  Proportion  of  RTOK  due  to  system-generated  detections 

e  Threshold  value  for  cannibalization  and  back-order 

Inventory  data 


MISSION 

of 

Rome  Air  Development  Center 

RAVC  plant,  and  executes  research,  development,  tut  and 
-selected  acquisition  programs  in  support  of  Command,  Control 
Communications  and  Intelligence.  (C5I )  activities.  Technical 
and  engineering  support  uiithin  areas  of  technical  competence 
is  provided  to  ESP  Program  Offices  [POs]  and  other  ESP 
elements.  The  principal  technical  mission  areas  are 
communications,  electromagnetic  guidance  and  control,  sur¬ 
veillance  o f  ground  and  aerospace  objects,  intelligence  data 
collection  and  handling,  information  system  technology, 
ionospheric  propagation,  solid  state  sciences,  microtmve 
physics  and  electronic  reliability,  maintainability  and 
compatibility. 


